[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 548366570 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002297] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004006] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008008] pid_max: default: 32768 minimum: 301 [ 0.010056] LSM: Security Framework initializing [ 0.011037] Yama: becoming mindful. [ 0.012027] SELinux: Initializing. [ 0.013060] *** VALIDATE selinux *** [ 0.020354] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024698] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026095] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027230] *** VALIDATE tmpfs *** [ 0.029176] *** VALIDATE proc *** [ 0.030332] *** VALIDATE cgroup *** [ 0.031013] *** VALIDATE cgroup2 *** [ 0.033310] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034164] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036060] Spectre V2 : User space: Vulnerable [ 0.037014] Speculative Store Bypass: Vulnerable [ 0.040237] debug: unmapping init [mem 0xffffffffb0259000-0xffffffffb0260fff] [ 0.043207] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045016] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046041] ... version: 2 [ 0.047012] ... bit width: 48 [ 0.048010] ... generic registers: 4 [ 0.049010] ... value mask: 0000ffffffffffff [ 0.050017] ... max period: 00007fffffffffff [ 0.051014] ... fixed-purpose events: 3 [ 0.052011] ... event mask: 000000070000000f [ 0.053443] rcu: Hierarchical SRCU implementation. [ 0.055777] smp: Bringing up secondary CPUs ... [ 0.056609] x86: Booting SMP configuration: [ 0.057018] .... node #0, CPUs: #1 #2 #3 [ 0.060276] smp: Brought up 1 node, 4 CPUs [ 0.062010] smpboot: Max logical packages: 1 [ 0.063014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.253348] node 0 deferred pages initialised in 189ms [ 0.256170] devtmpfs: initialized [ 0.257243] x86/mm: Memory block size: 128MB [ 0.259814] gcov: version magic: 0x41383552 [ 0.261310] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.262073] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.263331] pinctrl core: initialized pinctrl subsystem [ 0.264215] [ 0.264796] ************************************************************* [ 0.265010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.266009] ** ** [ 0.267008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.268009] ** ** [ 0.269007] ** This means that this kernel is built to expose internal ** [ 0.270007] ** IOMMU data structures, which may compromise security on ** [ 0.271009] ** your system. ** [ 0.272008] ** ** [ 0.273006] ** If you see this message and you are not debugging the ** [ 0.274007] ** kernel, report this immediately to your vendor! ** [ 0.275008] ** ** [ 0.276004] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.277008] ************************************************************* [ 0.278718] NET: Registered protocol family 16 [ 0.279356] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.280032] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.281030] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.282434] cpuidle: using governor menu [ 0.283658] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.285362] PCI: Using configuration type 1 for base access [ 0.286086] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.293085] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.295031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.298094] cryptd: max_cpu_qlen set to 1000 [ 0.300376] ACPI: Added _OSI(Module Device) [ 0.302013] ACPI: Added _OSI(Processor Device) [ 0.303009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.305011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.309000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.315437] ACPI: Interpreter enabled [ 0.316054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.317007] ACPI: Using IOAPIC for interrupt routing [ 0.319064] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.321341] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.327994] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.329024] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.332013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.334057] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.338150] acpiphp: Slot [2] registered [ 0.340067] acpiphp: Slot [3] registered [ 0.341066] acpiphp: Slot [4] registered [ 0.342073] acpiphp: Slot [5] registered [ 0.344111] acpiphp: Slot [6] registered [ 0.345057] acpiphp: Slot [7] registered [ 0.346069] acpiphp: Slot [8] registered [ 0.347053] acpiphp: Slot [9] registered [ 0.348142] acpiphp: Slot [10] registered [ 0.348900] acpiphp: Slot [11] registered [ 0.350068] acpiphp: Slot [12] registered [ 0.350997] acpiphp: Slot [13] registered [ 0.352053] acpiphp: Slot [14] registered [ 0.352895] acpiphp: Slot [15] registered [ 0.353039] acpiphp: Slot [16] registered [ 0.353682] acpiphp: Slot [17] registered [ 0.355044] acpiphp: Slot [18] registered [ 0.355816] acpiphp: Slot [19] registered [ 0.356044] acpiphp: Slot [20] registered [ 0.356700] acpiphp: Slot [21] registered [ 0.357039] acpiphp: Slot [22] registered [ 0.357793] acpiphp: Slot [23] registered [ 0.359059] acpiphp: Slot [24] registered [ 0.359812] acpiphp: Slot [25] registered [ 0.360054] acpiphp: Slot [26] registered [ 0.360907] acpiphp: Slot [27] registered [ 0.363066] acpiphp: Slot [28] registered [ 0.364040] acpiphp: Slot [29] registered [ 0.364858] acpiphp: Slot [30] registered [ 0.365039] acpiphp: Slot [31] registered [ 0.365813] PCI host bridge to bus 0000:00 [ 0.367013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.368008] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.369010] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.371011] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.372007] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.374013] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.375170] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.377652] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.380442] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.390680] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.394523] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.397012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.398012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.400013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.403436] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.405705] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.408030] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.410395] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.414013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.425859] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.430013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.435698] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.443019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.449011] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.466019] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.473823] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.482022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.488018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.503017] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.511309] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.518035] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.521026] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.541023] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.548393] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.557016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.563016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.576045] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.594182] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.601014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.612026] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.623045] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.636418] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.647021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.655020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.673020] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.684909] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.687349] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.689312] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.691305] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.693159] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.696174] iommu: Default domain type: Passthrough [ 0.697409] SCSI subsystem initialized [ 0.699096] ACPI: bus type USB registered [ 0.700106] usbcore: registered new interface driver usbfs [ 0.702057] usbcore: registered new interface driver hub [ 0.703051] usbcore: registered new device driver usb [ 0.705140] pps_core: LinuxPPS API ver. 1 registered [ 0.706006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.708056] PTP clock support registered [ 0.710127] EDAC MC: Ver: 3.0.0 [ 0.711513] PCI: Using ACPI for IRQ routing [ 0.712834] NetLabel: Initializing [ 0.713006] NetLabel: domain hash size = 128 [ 0.714030] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.715121] NetLabel: unlabeled traffic allowed by default [ 0.716136] vgaarb: loaded [ 0.717319] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.720013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.727006] clocksource: Switched to clocksource kvm-clock [ 0.827266] VFS: Disk quotas dquot_6.6.0 [ 0.829880] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.834661] *** VALIDATE ramfs *** [ 0.836511] *** VALIDATE hugetlbfs *** [ 0.837468] pnp: PnP ACPI init [ 0.839467] pnp: PnP ACPI: found 6 devices [ 0.854964] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.856954] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.858263] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.859612] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.861238] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.863478] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.866175] NET: Registered protocol family 2 [ 0.868523] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.871755] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.873807] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.877079] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.879016] TCP: Hash tables configured (established 65536 bind 65536) [ 0.880727] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.882548] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.884099] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.885939] NET: Registered protocol family 1 [ 0.888259] RPC: Registered named UNIX socket transport module. [ 0.890199] RPC: Registered udp transport module. [ 0.891610] RPC: Registered tcp transport module. [ 0.892881] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.894232] NET: Registered protocol family 44 [ 0.895333] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.896739] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.897744] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.898897] PCI: CLS 0 bytes, default 64 [ 0.900572] Unpacking initramfs... [ 2.260885] debug: unmapping init [mem 0xffff9a98bcc54000-0xffff9a98bffbffff] [ 2.264953] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.267363] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.270394] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.808030] Initialise system trusted keyrings [ 2.809374] Key type blacklist registered [ 2.810829] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.819921] zbud: loaded [ 2.824073] *** VALIDATE nfs *** [ 2.825489] *** VALIDATE nfs4 *** [ 2.827169] pstore: using deflate compression [ 2.832115] Platform Keyring initialized [ 2.979392] NET: Registered protocol family 38 [ 2.981353] Key type asymmetric registered [ 2.982975] Asymmetric key parser 'x509' registered [ 2.984957] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.988376] io scheduler mq-deadline registered [ 2.990160] io scheduler kyber registered [ 2.991765] io scheduler bfq registered [ 2.994446] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.000879] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.004173] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.007921] ACPI: Power Button [PWRF] [ 3.115833] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.216875] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.429448] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.520257] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.717043] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.747171] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.777147] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.782593] Non-volatile memory driver v1.3 [ 3.784466] Linux agpgart interface v0.103 [ 3.818589] virtio_blk virtio1: [vda] 67960 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.821348] vda: detected capacity change from 0 to 34795520 [ 3.841853] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.844698] vdb: detected capacity change from 0 to 1073741824 [ 3.858909] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.862056] vdc: detected capacity change from 0 to 2621440000 [ 3.874430] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.876818] vdd: detected capacity change from 0 to 2621440000 [ 3.889243] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.891696] vde: detected capacity change from 0 to 4294967296 [ 3.904140] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.906783] vdf: detected capacity change from 0 to 4294967296 [ 3.912756] libphy: Fixed MDIO Bus: probed [ 3.917264] usbcore: registered new interface driver usbserial_generic [ 3.919535] usbserial: USB Serial support registered for generic [ 3.921547] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.925087] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.926585] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.929170] mousedev: PS/2 mouse device common for all mice [ 3.931308] rtc_cmos 00:05: RTC can wake from S4 [ 3.936259] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.936954] rtc_cmos 00:05: registered as rtc0 [ 3.941094] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.943348] intel_pstate: CPU model not supported [ 3.946459] hid: raw HID events driver (C) Jiri Kosina [ 3.946833] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.948308] usbcore: registered new interface driver usbhid [ 3.948312] usbhid: USB HID core driver [ 3.948420] drop_monitor: Initializing network drop monitor service [ 3.948545] Initializing XFRM netlink socket [ 3.948893] NET: Registered protocol family 10 [ 3.950550] Segment Routing with IPv6 [ 3.956272] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.957442] NET: Registered protocol family 17 [ 3.965870] mpls_gso: MPLS GSO support [ 3.971510] RAS: Correctable Errors collector initialized. [ 3.973270] AVX version of gcm_enc/dec engaged. [ 3.974602] AES CTR mode by8 optimization enabled [ 4.094499] sched_clock: Marking stable (4094474407, 0)->(5048529768, -954055361) [ 4.097656] registered taskstats version 1 [ 4.105088] Loading compiled-in X.509 certificates [ 4.107052] zswap: loaded using pool lzo/zbud [ 4.186410] Key type big_key registered [ 4.198913] Key type encrypted registered [ 4.200609] ima: No TPM chip found, activating TPM-bypass! [ 4.204296] ima: Allocated hash algorithm: sha1 [ 4.205870] ima: No architecture policies found [ 4.207372] evm: Initialising EVM extended attributes: [ 4.208918] evm: security.selinux [ 4.209852] evm: security.ima [ 4.210742] evm: security.capability [ 4.211780] evm: HMAC attrs: 0x1 [ 4.213775] rtc_cmos 00:05: setting system clock to 2025-10-07 13:10:22 UTC (1759842622) [ 4.220117] debug: unmapping init [mem 0xffffffffb1203000-0xffffffffb13fffff] [ 4.223155] debug: unmapping init [mem 0xffffffffaff82000-0xffffffffb0258fff] [ 4.231321] Write protecting the kernel read-only data: 28672k [ 4.237520] debug: unmapping init [mem 0xffffffffae603000-0xffffffffae7fffff] [ 4.240630] debug: unmapping init [mem 0xffffffffaef14000-0xffffffffaeffffff] [ 4.281352] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.290601] systemd[1]: Detected virtualization kvm. [ 4.292396] systemd[1]: Detected architecture x86-64. [ 4.294147] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.332857] systemd[1]: No hostname configured. [ 4.334620] systemd[1]: Set hostname to . [ 4.336970] random: systemd: uninitialized urandom read (16 bytes read) [ 4.338812] systemd[1]: Initializing machine ID from random generator. [ 4.388525] random: ln: uninitialized urandom read (6 bytes read) [ 4.483271] random: systemd: uninitialized urandom read (16 bytes read) [ 4.485668] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.489930] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.499836] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.269633] device-mapper: uevent: version 1.0.3 [ 5.271427] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.445454] virtio_net virtio0 ens2: renamed from eth0 [ 6.536449] random: fast init done [ 7.152321] scsi host0: ata_piix [ 7.219301] scsi host1: ata_piix [ 7.220330] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.222206] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.722151] random: crng init done [ 11.723469] random: 7 urandom warning(s) missed due to ratelimiting [ 12.989401] dracut-initqueue[594]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 13.824654] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.283796] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.728083] SELinux: Disabled at runtime. [ 15.808226] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.819352] systemd[1]: Detected virtualization kvm. [ 15.820716] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.620136] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.624130] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.637386] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.643554] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.647224] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.653555] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.659617] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ 16.892580] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 17.398243] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.966551] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.014192] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 18.160845] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 18.190316] EDAC sbridge: Ver: 1.1.2 [ 21.374664] Key type dns_resolver registered [ 21.775660] NFS: Registering the id_resolver key type [ 21.778372] Key type id_resolver registered [ 21.781260] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg217-server login: [ 42.559765] libcfs: loading out-of-tree module taints kernel. [ 42.575224] alg: No test for adler32 (adler32-zlib) [ 43.326311] Key type ._llcrypt registered [ 43.327182] Key type .llcrypt registered [ 43.360153] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 49.599256] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 50.114992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 50.347058] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 50.572599] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 50.574400] LNet: Accept secure, port 988 [ 52.183125] Key type lgssc registered [ 52.686724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 58.819058] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 59.702527] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 62.997042] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 65.484099] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 68.053807] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 73.259564] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 77.678606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 77.707740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 78.797182] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 78.808574] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 78.849197] Lustre: lustre-MDT0000: new disk, initializing [ 78.875207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 78.882351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 80.349609] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 85.607148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 85.639493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 85.666602] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 85.675331] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 85.678198] Lustre: Skipped 1 previous similar message [ 85.716694] Lustre: lustre-MDT0001: new disk, initializing [ 85.740110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 85.748434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 85.751705] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 87.152196] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 92.265211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 92.293581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 92.298436] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 92.388395] Lustre: lustre-OST0000: new disk, initializing [ 92.390404] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 92.406700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 93.836637] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 94.186763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 94.190478] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 98.899610] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 98.933068] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 98.935441] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 98.970525] Lustre: lustre-OST0001: new disk, initializing [ 98.972515] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 98.988676] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 100.347177] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 102.697201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 102.699477] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 105.924070] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 108.406941] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 114.615475] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing check_logdir /tmp/testlogs/ [ 116.158116] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing yml_node [ 117.524081] Lustre: DEBUG MARKER: Client: 2.15.7.3 [ 118.388239] Lustre: DEBUG MARKER: MDS: 2.15.7.3 [ 119.367823] Lustre: DEBUG MARKER: OSS: 2.15.7.3 [ 119.972915] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Oct 7 09:12:17 EDT 2025 [ 122.796865] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball 110 [ 123.308943] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 137.183462] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 137.186381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 137.189974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 138.719734] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 138.719874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 138.723396] Lustre: Skipped 2 previous similar messages [ 138.726871] Lustre: Skipped 3 previous similar messages [ 140.989309] Lustre: server umount lustre-MDT0000 complete [ 142.251860] LustreError: 7232:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759842760 with bad export cookie 17378733205963899903 [ 142.253269] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 142.255312] LustreError: 7232:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 143.839783] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 143.839968] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 143.840210] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 143.843336] Lustre: Skipped 1 previous similar message [ 143.847016] LustreError: Skipped 1 previous similar message [ 147.871936] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 147.872027] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 147.873890] Lustre: Skipped 2 previous similar messages [ 147.879371] LustreError: Skipped 1 previous similar message [ 148.393842] Lustre: server umount lustre-MDT0001 complete [ 164.319150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 164.330309] Lustre: server umount lustre-OST0000 complete [ 171.793551] Lustre: server umount lustre-OST0001 complete [ 174.045956] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 176.922996] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 180.601635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 182.817712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 184.801593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 186.931411] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 189.313601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 189.337583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 189.427597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 189.439287] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 189.478388] Lustre: lustre-MDT0000: new disk, initializing [ 189.511786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 189.521046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 191.103640] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 195.388508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 195.412415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 195.447295] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 195.542483] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 196.984345] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 201.028150] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 203.016475] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 205.382960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 205.409181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 205.412405] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 205.505622] Lustre: lustre-OST0000: new disk, initializing [ 205.508075] Lustre: Skipped 1 previous similar message [ 205.510785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 205.513946] Lustre: Skipped 2 previous similar messages [ 205.541888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 205.544688] Lustre: Skipped 1 previous similar message [ 206.998683] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 207.158698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 207.162322] Lustre: Skipped 1 previous similar message [ 207.164683] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 210.930829] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 212.475117] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 212.559133] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 214.051906] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 214.133665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 217.569458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 217.569811] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 217.574717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 217.577548] LustreError: Skipped 1 previous similar message [ 227.808797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 227.810854] Lustre: Skipped 4 previous similar messages [ 229.343107] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 229.401083] Lustre: server umount lustre-OST0000 complete [ 231.391799] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 231.394593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 231.398429] Lustre: Skipped 1 previous similar message [ 237.998204] LustreError: 17989:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759842856 with bad export cookie 17378733205963901345 [ 238.000482] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 238.003229] LustreError: 17989:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 238.116561] Lustre: server umount lustre-MDT0001 complete [ 238.117774] Lustre: Skipped 1 previous similar message [ 239.431686] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:14:17 (1759842857) [ 240.244348] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 240.806259] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:14:18 (1759842858) [ 243.060325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 243.213681] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 243.235969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 244.745148] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 247.682447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 249.193050] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 251.675482] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 253.613883] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 255.906619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 257.443688] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 260.061407] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 268.256959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 268.257517] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 268.260697] Lustre: Skipped 1 previous similar message [ 268.263619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 268.266383] Lustre: Skipped 4 previous similar messages [ 272.470827] Lustre: server umount lustre-OST0000 complete [ 278.496473] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 288.223176] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 289.643848] LustreError: 18838:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759842907 with bad export cookie 17378733205963902199 [ 289.646578] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 289.648393] LustreError: 18838:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 294.882801] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 295.860774] Key type lgssc unregistered [ 295.981350] LNet: 22510:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 295.984141] LNet: Removed LNI 192.168.202.117@tcp [ 296.320799] Key type .llcrypt unregistered [ 296.322090] Key type ._llcrypt unregistered [ 305.037190] alg: No test for adler32 (adler32-zlib) [ 305.788367] Key type ._llcrypt registered [ 305.789650] Key type .llcrypt registered [ 305.825742] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 306.185827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 306.400022] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 306.473401] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 306.476404] LNet: Accept secure, port 988 [ 308.055183] Key type lgssc registered [ 308.393611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 310.938582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 312.066910] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 312.080795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 313.459609] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 316.121820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 316.230490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 317.495180] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 319.812501] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 321.589921] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 323.778413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 323.885916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 325.220854] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 326.948661] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 327.673312] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 336.353139] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 336.353505] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 336.356387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 336.359365] Lustre: Skipped 1 previous similar message [ 336.362806] Lustre: Skipped 1 previous similar message [ 341.474126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 341.476084] Lustre: Skipped 1 previous similar message [ 342.289445] Lustre: server umount lustre-OST0000 complete [ 346.592432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 346.592976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 346.596886] Lustre: Skipped 1 previous similar message [ 349.633370] Lustre: server umount lustre-MDT0000 complete [ 350.883379] LustreError: 25919:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759842969 with bad export cookie 1427583811549868816 [ 350.885362] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 350.887089] LustreError: 25919:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 350.990852] Lustre: server umount lustre-MDT0001 complete [ 355.655077] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 356.602664] Key type lgssc unregistered [ 356.706451] LNet: 27624:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.709806] LNet: Removed LNI 192.168.202.117@tcp [ 357.003820] Key type .llcrypt unregistered [ 357.004827] Key type ._llcrypt unregistered [ 365.705944] alg: No test for adler32 (adler32-zlib) [ 366.457356] Key type ._llcrypt registered [ 366.458400] Key type .llcrypt registered [ 366.493473] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 366.873164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 367.030906] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 367.111566] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 367.113530] LNet: Accept secure, port 988 [ 368.687160] Key type lgssc registered [ 369.020867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 371.582990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 372.716760] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.731339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 374.155581] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 377.076485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 377.208464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 378.639588] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 381.224193] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 383.187110] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 385.539023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 385.653434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 387.124420] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 388.708252] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 389.664126] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 397.280593] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 397.281876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 397.284033] Lustre: Skipped 2 previous similar messages [ 397.286860] Lustre: Skipped 2 previous similar messages [ 401.012341] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.17@tcp (stopping) [ 402.400354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 402.402394] Lustre: Skipped 2 previous similar messages [ 402.759680] Lustre: server umount lustre-MDT0000 complete [ 403.993922] LustreError: 29066:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843022 with bad export cookie 4502613781389935458 [ 403.996334] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 403.997397] LustreError: 29066:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 406.123475] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 406.129291] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.17@tcp (stopping) [ 407.519882] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 407.520093] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 410.592160] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 410.592153] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 410.594018] Lustre: Skipped 2 previous similar messages [ 410.597876] LustreError: Skipped 1 previous similar message [ 415.711666] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 415.716858] LustreError: Skipped 2 previous similar messages [ 418.271144] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 418.358141] Lustre: server umount lustre-MDT0001 complete [ 430.186991] Lustre: server umount lustre-OST0000 complete [ 436.781291] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 437.766407] Key type lgssc unregistered [ 437.885360] LNet: 33061:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 437.888539] LNet: Removed LNI 192.168.202.117@tcp [ 438.168204] Key type .llcrypt unregistered [ 438.169213] Key type ._llcrypt unregistered [ 447.186270] alg: No test for adler32 (adler32-zlib) [ 447.937443] Key type ._llcrypt registered [ 447.938436] Key type .llcrypt registered [ 447.972683] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 448.392319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 448.518875] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 448.589054] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 448.591385] LNet: Accept secure, port 988 [ 450.167118] Key type lgssc registered [ 450.481865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 452.964675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 454.090688] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 454.104144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 455.353563] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 458.075597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 458.178863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 459.557700] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 462.028411] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 463.804350] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 465.920884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 466.019950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 467.358597] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 469.091626] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 469.733546] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 478.691971] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 478.692152] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 478.692767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 478.699883] Lustre: Skipped 1 previous similar message [ 481.397626] Lustre: lustre-OST0000: Not available for connect from 192.168.202.17@tcp (stopping) [ 481.399618] Lustre: Skipped 1 previous similar message [ 482.958920] Lustre: server umount lustre-OST0000 complete [ 499.168066] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 499.171430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 499.175396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 500.192955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 501.770617] Lustre: server umount lustre-MDT0000 complete [ 502.999040] LustreError: 34500:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843121 with bad export cookie 6318439246730030450 [ 503.000978] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 503.002881] LustreError: 34500:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 509.171680] Lustre: server umount lustre-MDT0001 complete [ 513.867918] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 514.910901] Key type lgssc unregistered [ 515.039447] LNet: 38345:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 515.043698] LNet: Removed LNI 192.168.202.117@tcp [ 515.343225] Key type .llcrypt unregistered [ 515.344858] Key type ._llcrypt unregistered [ 524.563949] alg: No test for adler32 (adler32-zlib) [ 525.315365] Key type ._llcrypt registered [ 525.316520] Key type .llcrypt registered [ 525.361104] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 525.726113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 525.984471] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 526.069434] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 526.071167] LNet: Accept secure, port 988 [ 527.655072] Key type lgssc registered [ 528.002793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 530.732730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 531.875353] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.890590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 533.316301] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 536.271378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 536.407256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 537.892780] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 540.589196] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 542.639109] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 544.936933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 545.056658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 546.547893] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 547.110195] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 549.258664] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 556.238905] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 557.453322] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 557.795392] LNetError: 120-3: Refusing connection from 192.168.202.117 for 127.0.0.2@tcp: No matching NI [ 557.798184] LNetError: 39108:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 557.801771] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 568.182223] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-10-07 9h19m45s [ 568.692347] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 570.996160] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.17@tcp (stopping) [ 571.873195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 571.875569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 571.878038] Lustre: Skipped 2 previous similar messages [ 571.879254] LNetError: 120-3: Refusing connection from 192.168.202.117 for 127.0.0.2@tcp: No matching NI [ 571.880323] Lustre: Skipped 2 previous similar messages [ 571.882066] LNetError: 39105:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 571.888867] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 575.620586] Lustre: server umount lustre-MDT0000 complete [ 576.107401] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 576.793186] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 576.994569] LNetError: 120-3: Refusing connection from 192.168.202.117 for 127.0.0.2@tcp: No matching NI [ 576.997158] LNetError: 39106:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 577.000688] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 579.112408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 579.151591] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 579.153240] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 579.153265] LustreError: 11-0: MGC192.168.202.117@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 579.155848] LustreError: Skipped 5 previous similar messages [ 579.160876] Lustre: Evicted from MGS (at 192.168.202.117@tcp) after server handle changed from 0x193aabb315439aec to 0x193aabb315439f37 [ 579.170898] Lustre: MGC192.168.202.117@tcp: Connection restored to 192.168.202.117@tcp (at 0@lo) [ 579.253841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 579.282094] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 580.707575] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 581.852707] Lustre: DEBUG MARKER: Wait for df (28794) ... [ 582.441869] Lustre: DEBUG MARKER: done [ 584.673038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 584.674177] LNetError: 120-3: Refusing connection from 192.168.202.117 for 127.0.0.2@tcp: No matching NI [ 584.674386] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 584.675951] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 584.678275] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 584.678437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 584.680930] LNetError: 39107:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 584.692812] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 589.650337] Lustre: server umount lustre-OST0000 complete [ 594.912539] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 594.912698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 594.920307] Lustre: Skipped 1 previous similar message [ 594.922717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 594.923863] LNetError: 120-3: Refusing connection from 192.168.202.117 for 127.0.0.2@tcp: No matching NI [ 594.924974] Lustre: Skipped 2 previous similar messages [ 594.931092] LNetError: 39108:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 594.933597] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 597.064081] Lustre: server umount lustre-MDT0000 complete [ 598.335483] LustreError: 39786:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843216 with bad export cookie 1817954185241665335 [ 598.338556] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 598.339256] LustreError: 39786:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 603.184719] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 604.147298] Key type lgssc unregistered [ 604.248324] LNet: 44893:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 604.251616] LNet: Removed LNI 192.168.202.117@tcp [ 604.544538] Key type .llcrypt unregistered [ 604.545702] Key type ._llcrypt unregistered [ 616.107656] alg: No test for adler32 (adler32-zlib) [ 616.859363] Key type ._llcrypt registered [ 616.860783] Key type .llcrypt registered [ 616.897850] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 622.920657] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 623.389246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 623.541639] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 623.622570] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 623.624323] LNet: Accept secure, port 988 [ 625.207113] Key type lgssc registered [ 625.545956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 628.540274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 630.876812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 632.994678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 635.181971] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 637.511356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 637.540826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 638.624050] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 638.634830] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 638.675144] Lustre: lustre-MDT0000: new disk, initializing [ 638.699693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 638.707664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 640.230324] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 644.503799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 644.532308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 644.569893] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 644.589140] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 644.591680] Lustre: Skipped 1 previous similar message [ 644.629058] Lustre: lustre-MDT0001: new disk, initializing [ 644.658635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 644.670717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 644.674512] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 646.206990] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 650.085265] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 652.100827] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 654.343557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 654.367520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 654.369975] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 654.464040] Lustre: lustre-OST0000: new disk, initializing [ 654.466199] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 654.490933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 655.898617] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 659.678039] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 661.142410] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 662.515319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 662.518396] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 663.263571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 664.743661] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 664.824199] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 667.618577] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 667.618931] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 667.621124] LustreError: Skipped 1 previous similar message [ 667.626991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 667.630891] Lustre: Skipped 1 previous similar message [ 671.585871] Lustre: server umount lustre-OST0000 complete [ 675.295889] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 675.298354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 675.301845] Lustre: Skipped 1 previous similar message [ 675.303480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 677.855862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 677.856381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 678.978623] Lustre: server umount lustre-MDT0000 complete [ 680.253111] LustreError: 51064:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843298 with bad export cookie 7754106691129271132 [ 680.256766] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 680.257768] LustreError: 51064:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 680.397311] Lustre: server umount lustre-MDT0001 complete [ 683.235159] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 09:21:40 (1759843300) [ 685.512464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 685.654226] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 685.674608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 687.077929] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 689.851599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 689.959175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 691.386628] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 693.862401] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 695.846989] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 698.172252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 698.282334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 699.711655] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 702.343739] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 709.565380] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-10-07 9h22m07s [ 710.146803] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 712.464202] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 713.309300] Lustre: DEBUG MARKER: Creating a test file and stat it [ 713.328334] LustreError: 53538:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 sleeping for 40000ms [ 719.331971] LNetError: 120-3: Refusing connection from 192.168.202.117 for 127.0.0.2@tcp: No matching NI [ 719.334391] LNetError: 47197:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 719.337114] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 733.814647] Lustre: lustre-MDT0000: Client 9c67123d-cc5a-4603-9e34-23571e29ff34 (at 192.168.202.17@tcp) reconnecting [ 753.375151] LustreError: 53538:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 awake [ 754.154521] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 754.998950] Lustre: DEBUG MARKER: done [ 756.705350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 756.706238] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 756.706627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 756.706632] Lustre: Skipped 1 previous similar message [ 756.709286] Lustre: Skipped 1 previous similar message [ 761.825121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 761.827852] Lustre: Skipped 2 previous similar messages [ 762.710807] Lustre: server umount lustre-OST0000 complete [ 766.944721] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 766.944900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 766.951754] Lustre: Skipped 1 previous similar message [ 766.953935] LNetError: 120-3: Refusing connection from 192.168.202.117 for 127.0.0.2@tcp: No matching NI [ 766.956234] LNetError: 47195:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 766.958580] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 770.117398] Lustre: server umount lustre-MDT0000 complete [ 771.365088] LustreError: 53524:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843389 with bad export cookie 7754106691129271972 [ 771.367825] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 771.368059] LustreError: 53524:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 776.315385] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 777.268082] Key type lgssc unregistered [ 777.378286] LNet: 57949:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 777.380940] LNet: Removed LNI 192.168.202.117@tcp [ 777.665379] Key type .llcrypt unregistered [ 777.667144] Key type ._llcrypt unregistered [ 788.598243] alg: No test for adler32 (adler32-zlib) [ 789.350379] Key type ._llcrypt registered [ 789.351578] Key type .llcrypt registered [ 789.385813] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 795.149422] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 795.638821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 795.779108] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 795.868806] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 795.870467] LNet: Accept secure, port 988 [ 797.447058] Key type lgssc registered [ 797.794161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 800.807658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 803.024277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 805.043053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 807.162353] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 809.483519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 809.505570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 810.598143] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 810.608414] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 810.649037] Lustre: lustre-MDT0000: new disk, initializing [ 810.675421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 810.682904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 812.191581] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 816.494989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 816.524341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 816.561750] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 816.582087] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 816.585152] Lustre: Skipped 1 previous similar message [ 816.623637] Lustre: lustre-MDT0001: new disk, initializing [ 816.652877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 816.664171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 816.667595] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 818.132471] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 821.968290] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 823.917849] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 826.212643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 826.239656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 826.243934] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 826.338763] Lustre: lustre-OST0000: new disk, initializing [ 826.340951] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 826.367560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 827.125032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 827.128288] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 827.781992] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 831.581753] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 833.067835] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 833.141432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 834.587166] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 834.664038] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 837.601759] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 837.602531] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 837.603371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 837.604040] LustreError: Skipped 1 previous similar message [ 837.607514] Lustre: Skipped 1 previous similar message [ 841.369053] Lustre: server umount lustre-OST0000 complete [ 842.719986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 842.724208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 842.725940] Lustre: Skipped 1 previous similar message [ 847.327710] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 847.330951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 847.334847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 848.708047] Lustre: server umount lustre-MDT0000 complete [ 849.988810] LustreError: 65513:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843468 with bad export cookie 11727523663712745180 [ 849.991295] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 849.992293] LustreError: 65513:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 850.112052] Lustre: server umount lustre-MDT0001 complete [ 852.991773] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 09:24:30 (1759843470) [ 853.526749] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 854.153677] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 09:24:31 (1759843471) [ 858.003525] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 859.426487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 859.565357] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 859.578253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 859.915247] Lustre: Failing over lustre-MDT0000 [ 859.917606] LustreError: 67048:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 859.921881] LustreError: 67048:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 859.925554] LustreError: 67048:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 866.271113] Lustre: 67099:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759843478/real 1759843478] req@00000000313c76b6 x1845329567695552/t0(0) o251->MGC192.168.202.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1759843484 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 866.332373] Lustre: server umount lustre-MDT0000 complete [ 868.533879] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:24:46 (1759843486) [ 870.722433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 870.857490] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 870.869913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 872.283074] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 875.179232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 876.777910] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 879.257762] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 881.144068] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 883.367066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 883.469162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 883.471038] Lustre: Skipped 1 previous similar message [ 884.900175] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 887.436263] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 894.657805] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 901.089634] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 901.090156] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 901.090703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 901.090708] Lustre: Skipped 1 previous similar message [ 901.094153] Lustre: Skipped 2 previous similar messages [ 902.600286] Lustre: server umount lustre-MDT0000 complete [ 903.879106] LustreError: 67646:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843522 with bad export cookie 11727523663712746293 [ 903.881671] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 903.884319] LustreError: 67646:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 905.132801] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 906.664690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 909.487768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 910.240482] Lustre: Evicted from MGS (at 192.168.202.117@tcp) after server handle changed from 0xa2c088a31695fb35 to 0xa2c088a3169602ba [ 910.244564] Lustre: MGC192.168.202.117@tcp: Connection restored to 192.168.202.117@tcp (at 0@lo) [ 910.346178] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 910.369275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 911.392617] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 911.732094] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 913.439205] Lustre: 60267:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759843524/real 1759843524] req@0000000096e3acae x1845329567710208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759843531 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 914.546377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 914.692404] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:65 [ 916.081106] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 916.447443] Lustre: 60267:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759843526/real 1759843526] req@000000004bc67f19 x1845329567710272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759843533 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 917.732047] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 918.534463] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 920.486262] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 923.105944] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 923.107273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 923.109608] Lustre: Skipped 1 previous similar message [ 923.113974] Lustre: Skipped 4 previous similar messages [ 928.388362] Lustre: server umount lustre-MDT0000 complete [ 928.390678] Lustre: Skipped 1 previous similar message [ 929.660527] LustreError: 67644:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843547 with bad export cookie 11727523663712748218 [ 929.663013] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 929.665775] LustreError: 67644:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 931.959310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 935.091610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 935.392575] Lustre: Evicted from MGS (at 192.168.202.117@tcp) after server handle changed from 0xa2c088a3169602ba to 0xa2c088a3169608c5 [ 935.396273] Lustre: MGC192.168.202.117@tcp: Connection restored to (at 0@lo) [ 935.397674] Lustre: Skipped 1 previous similar message [ 935.499594] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 935.504054] LustreError: Skipped 2 previous similar messages [ 935.535504] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 936.891248] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 939.692524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 939.855737] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:97 [ 940.815113] Lustre: 60266:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759843551/real 1759843551] req@00000000fc16aa3c x1845329567721216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759843558 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 940.822594] Lustre: 60266:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 941.331584] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 943.902711] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 945.125733] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 945.886444] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 949.216789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 949.217447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 949.218505] Lustre: Skipped 4 previous similar messages [ 949.224250] Lustre: Skipped 3 previous similar messages [ 953.734496] Lustre: server umount lustre-MDT0000 complete [ 953.735889] Lustre: Skipped 1 previous similar message [ 954.336314] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 954.342944] LustreError: Skipped 4 previous similar messages [ 954.974961] LustreError: 67646:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843573 with bad export cookie 11727523663712749765 [ 954.977934] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 954.978167] LustreError: 67646:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 957.215603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 958.334086] Lustre: DEBUG MARKER: files compared the same [ 965.087205] Lustre: 76789:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759843577/real 1759843577] req@00000000ff5ccbaf x1845329567730304/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759843583 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 965.098194] Lustre: 76789:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 971.847118] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 972.848432] Key type lgssc unregistered [ 972.956385] LNet: 77696:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 972.959882] LNet: Removed LNI 192.168.202.117@tcp [ 973.243418] Key type .llcrypt unregistered [ 973.244559] Key type ._llcrypt unregistered [ 982.435671] alg: No test for adler32 (adler32-zlib) [ 983.187314] Key type ._llcrypt registered [ 983.188297] Key type .llcrypt registered [ 983.225076] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 983.661257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 983.778520] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 983.858642] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 983.860540] LNet: Accept secure, port 988 [ 985.439117] Key type lgssc registered [ 985.781521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 988.389495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 989.562619] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 989.578847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 991.324902] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug +malloc all 8 [ 994.598051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 994.722135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 996.775659] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug +malloc all 8 [ 999.872891] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1002.399681] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1005.455339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1005.590675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1007.411136] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug +malloc all 8 [ 1010.634236] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1010.721483] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:129 [ 1018.915575] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 1026.015763] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1026.024549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1026.047655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1029.087987] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1029.089163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1031.604065] Lustre: server umount lustre-OST0000 complete [ 1034.209853] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1034.214929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1034.215691] Lustre: Skipped 1 previous similar message [ 1034.219566] Lustre: Skipped 2 previous similar messages [ 1039.327878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1039.776254] Lustre: server umount lustre-MDT0000 complete [ 1041.979319] LustreError: 79137:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843660 with bad export cookie 15137914900278049181 [ 1041.979896] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1041.985710] LustreError: 79137:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1042.157834] Lustre: server umount lustre-MDT0001 complete [ 1047.840678] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 1048.869772] Key type lgssc unregistered [ 1048.975396] LNet: 82831:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1048.978303] LNet: Removed LNI 192.168.202.117@tcp [ 1049.260220] Key type .llcrypt unregistered [ 1049.261221] Key type ._llcrypt unregistered [ 1058.035547] alg: No test for adler32 (adler32-zlib) [ 1058.787545] Key type ._llcrypt registered [ 1058.788970] Key type .llcrypt registered [ 1058.831582] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 1059.282211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1059.522695] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 1059.599032] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 1059.600366] LNet: Accept secure, port 988 [ 1061.175123] Key type lgssc registered [ 1061.496397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1064.028120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1081.503317] LustreError: 84263:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.117@tcp: failed processing log, type 1: rc = -5 [ 1112.223219] LustreError: 84263:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.117@tcp: failed processing log, type 4: rc = -110 [ 1142.943220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1144.331224] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1146.774805] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1149.227067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1149.264510] Lustre: *** cfs_fail_loc=706, val=0*** [ 1164.801227] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1164.824326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1164.844134] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 1166.136386] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1168.842109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1168.935690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1168.962218] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:161 [ 1170.300350] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1172.729482] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1174.488299] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1179.104760] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1179.106292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1179.114934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1179.117516] Lustre: Skipped 1 previous similar message [ 1181.334252] Lustre: server umount lustre-OST0000 complete [ 1184.224485] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1184.224516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1184.231103] Lustre: Skipped 1 previous similar message [ 1184.233021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1184.235099] Lustre: Skipped 1 previous similar message [ 1188.803212] Lustre: server umount lustre-MDT0000 complete [ 1189.344159] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1189.350981] LustreError: Skipped 1 previous similar message [ 1190.115320] LustreError: 86100:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843808 with bad export cookie 10582896736314245118 [ 1190.117984] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1190.118987] LustreError: 86100:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1190.254050] Lustre: server umount lustre-MDT0001 complete [ 1194.013939] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 1195.062505] Key type lgssc unregistered [ 1195.187444] LNet: 88003:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1195.190184] LNet: Removed LNI 192.168.202.117@tcp [ 1195.514348] Key type .llcrypt unregistered [ 1195.515920] Key type ._llcrypt unregistered [ 1205.488236] alg: No test for adler32 (adler32-zlib) [ 1206.240309] Key type ._llcrypt registered [ 1206.241524] Key type .llcrypt registered [ 1206.280231] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 1206.605331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1206.828693] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 1206.907280] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 1206.909144] LNet: Accept secure, port 988 [ 1208.495143] Key type lgssc registered [ 1208.825348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1211.370425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1213.959354] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1217.041770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1217.267903] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1217.314605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1218.777133] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1221.595558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1221.692533] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1221.698824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1221.699022] LustreError: Skipped 3 previous similar messages [ 1222.752334] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1222.755947] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:193 [ 1222.757772] LustreError: Skipped 2 previous similar messages [ 1223.073461] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1226.034279] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1226.038205] LustreError: Skipped 2 previous similar messages [ 1226.118389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1226.149532] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:193 [ 1227.715367] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1236.450339] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1236.452111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1236.462225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1236.463854] Lustre: Skipped 1 previous similar message [ 1241.571150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1241.575375] Lustre: Skipped 1 previous similar message [ 1242.245097] Lustre: server umount lustre-OST0000 complete [ 1246.690203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1246.690480] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1246.691788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1246.697682] Lustre: Skipped 2 previous similar messages [ 1250.263668] Lustre: server umount lustre-MDT0000 complete [ 1251.687974] Lustre: server umount lustre-MDT0001 complete [ 1259.487225] Lustre: 92873:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759843871/real 1759843871] req@00000000873e635f x1845329998654080/t0(0) o251->MGC192.168.202.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1759843877 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1259.511836] Lustre: server umount MGS complete [ 1263.404119] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:31:21 (1759843881) [ 1273.594001] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 1276.741617] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 1279.837170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1282.735291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1285.143286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1287.267311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1289.827680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1289.868476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1291.656721] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1296.077465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1296.118147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1296.219312] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1296.236573] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1296.275033] Lustre: lustre-MDT0001: new disk, initializing [ 1296.309219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1296.317515] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1296.321129] LustreError: 97567:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osd get update log failed: rc = -115 [ 1297.856832] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1302.293863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1302.326297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1302.329086] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1302.416419] Lustre: lustre-OST0000: new disk, initializing [ 1302.418750] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1302.442751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1303.873613] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1307.552296] Lustre: 98867:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@00000000eabf4978 x1845329998658560/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1759843937 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 1308.149930] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1310.691702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1310.712447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1310.762807] Lustre: lustre-MDT0000: new disk, initializing [ 1310.812894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1310.824273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1311.841126] Lustre: 99421:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@00000000873e635f x1845329998661568/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1759843942 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 1312.697135] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1316.642308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1316.645374] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1337.312531] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1337.317710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1337.326415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1337.329943] Lustre: Skipped 1 previous similar message [ 1341.410959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1341.411937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1341.412342] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1342.813858] Lustre: server umount lustre-OST0000 complete [ 1346.529124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1346.529266] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1346.532505] Lustre: Skipped 1 previous similar message [ 1351.648899] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1351.655393] LustreError: Skipped 1 previous similar message [ 1357.943996] Lustre: server umount lustre-MDT0001 complete [ 1357.945766] Lustre: Skipped 1 previous similar message [ 1365.471140] Lustre: 100612:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759843977/real 1759843977] req@000000009399d9b3 x1845329998670976/t0(0) o251->MGC192.168.202.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1759843983 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1369.193623] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:33:06 (1759843986) [ 1376.316218] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 1377.367550] Key type lgssc unregistered [ 1377.481468] LNet: 102017:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1377.486054] LNet: Removed LNI 192.168.202.117@tcp [ 1377.839262] Key type .llcrypt unregistered [ 1377.840780] Key type ._llcrypt unregistered [ 1384.366297] alg: No test for adler32 (adler32-zlib) [ 1385.118342] Key type ._llcrypt registered [ 1385.119352] Key type .llcrypt registered [ 1385.158981] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 1385.465820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1385.725080] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 1385.804341] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 1385.805881] LNet: Accept secure, port 988 [ 1387.383118] Key type lgssc registered [ 1387.707469] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1390.306199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1391.466855] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1391.482560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1392.889107] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1395.665441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1395.784241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1397.131975] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1399.564733] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1401.428567] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1403.641811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1403.757663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1405.074843] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1406.820724] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 1407.583732] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1416.163121] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1416.163474] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1416.166034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1416.169042] Lustre: Skipped 1 previous similar message [ 1416.171893] Lustre: Skipped 1 previous similar message [ 1421.284110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1421.289056] Lustre: Skipped 1 previous similar message [ 1421.386474] Lustre: server umount lustre-OST0000 complete [ 1426.401849] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1426.402158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1426.402936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1426.419745] Lustre: Skipped 1 previous similar message [ 1429.125119] Lustre: server umount lustre-MDT0000 complete [ 1430.369489] LustreError: 103277:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844048 with bad export cookie 4504992615313257692 [ 1430.373298] LustreError: 103277:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1430.373328] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1430.519093] Lustre: server umount lustre-MDT0001 complete [ 1435.304480] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 1436.258105] Key type lgssc unregistered [ 1436.363274] LNet: 106951:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1436.366516] LNet: Removed LNI 192.168.202.117@tcp [ 1436.653553] Key type .llcrypt unregistered [ 1436.654960] Key type ._llcrypt unregistered [ 1442.869287] alg: No test for adler32 (adler32-zlib) [ 1443.621371] Key type ._llcrypt registered [ 1443.622483] Key type .llcrypt registered [ 1443.658674] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 1444.099500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1444.209020] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 1444.283854] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 1444.285967] LNet: Accept secure, port 988 [ 1445.863081] Key type lgssc registered [ 1446.186089] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1448.186964] LustreError: 108078:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 1448.187927] LustreError: 108079:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 1448.191514] LustreError: 108078:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4997 [ 1448.193549] LustreError: 108078:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1448.197110] LustreError: 108078:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1448.198051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1449.362764] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1449.377560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1451.820549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1451.925047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1453.288865] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1454.924601] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing lsmod [ 1455.861183] LustreError: 109387:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 1455.862317] LustreError: 109388:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 1455.865985] LustreError: 109387:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4999 [ 1455.869184] LustreError: 109387:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1455.872335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1455.873651] LustreError: 109387:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1455.991742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1456.991626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1456.993288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1456.996125] Lustre: Skipped 2 previous similar messages [ 1456.998589] Lustre: Skipped 1 previous similar message [ 1462.240792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1462.243267] Lustre: Skipped 3 previous similar messages [ 1463.041096] Lustre: server umount lustre-MDT0000 complete [ 1464.248466] LustreError: 109574:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844082 with bad export cookie 6698963992915272036 [ 1464.250532] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1464.252617] LustreError: 109574:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1464.360314] Lustre: server umount lustre-MDT0001 complete [ 1470.943159] Lustre: 109979:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759844083/real 1759844083] req@00000000a1084c36 x1845330247163584/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759844089 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1471.037105] Lustre: server umount lustre-OST0000 complete [ 1473.800669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1473.952976] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1473.977494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1475.461334] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1478.268272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1478.384436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1479.792209] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1482.243797] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1484.104811] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1486.324962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1486.426793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1487.975765] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1489.516680] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 1490.854755] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1499.106153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1499.108411] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1499.109336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1499.112582] Lustre: Skipped 1 previous similar message [ 1504.225302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1504.226948] Lustre: Skipped 2 previous similar messages [ 1504.340059] Lustre: server umount lustre-OST0000 complete [ 1509.344595] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1509.344637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1509.345153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1509.354128] Lustre: Skipped 1 previous similar message [ 1511.758112] Lustre: server umount lustre-MDT0000 complete [ 1512.979109] LustreError: 110433:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844131 with bad export cookie 6698963992915272694 [ 1512.982548] LustreError: 110433:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1512.982890] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1517.902227] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 1518.933732] Key type lgssc unregistered [ 1519.037299] LNet: 114104:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1519.039870] LNet: Removed LNI 192.168.202.117@tcp [ 1519.371953] Key type .llcrypt unregistered [ 1519.373249] Key type ._llcrypt unregistered [ 1528.698315] alg: No test for adler32 (adler32-zlib) [ 1529.449419] Key type ._llcrypt registered [ 1529.450495] Key type .llcrypt registered [ 1529.495607] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 1529.827712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1530.102831] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 1530.179635] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 1530.181711] LNet: Accept secure, port 988 [ 1531.759153] Key type lgssc registered [ 1532.079805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1534.564428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1535.715479] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1535.729623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1537.081322] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1539.730408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1539.841718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1541.182962] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1543.570145] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1545.517079] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1547.736857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1547.838743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1549.088983] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1550.949235] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 1551.585543] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1558.548443] Lustre: Setting parameter lustre-client.llite.some_wrong_param in log lustre-client [ 1559.329107] Lustre: Modifying parameter lustre-client.llite.some_wrong_param in log lustre-client [ 1560.544893] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1560.547673] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1560.549863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1560.553763] Lustre: Skipped 1 previous similar message [ 1560.556720] Lustre: Skipped 1 previous similar message [ 1565.664969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1565.666832] Lustre: Skipped 1 previous similar message [ 1566.360684] Lustre: server umount lustre-OST0000 complete [ 1570.785069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1570.785128] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1570.785501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1570.788133] Lustre: Skipped 1 previous similar message [ 1573.635882] Lustre: server umount lustre-MDT0000 complete [ 1574.861921] LustreError: 115544:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844193 with bad export cookie 16382868362208663987 [ 1574.865202] LustreError: 115544:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1574.865663] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1574.986778] Lustre: server umount lustre-MDT0001 complete [ 1579.845215] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 1580.702696] Key type lgssc unregistered [ 1580.796289] LNet: 119321:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1580.798860] LNet: Removed LNI 192.168.202.117@tcp [ 1581.075069] Key type .llcrypt unregistered [ 1581.076223] Key type ._llcrypt unregistered [ 1587.561937] alg: No test for adler32 (adler32-zlib) [ 1588.314350] Key type ._llcrypt registered [ 1588.315371] Key type .llcrypt registered [ 1588.353166] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 1588.860765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1589.010284] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 1589.092040] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 1589.094200] LNet: Accept secure, port 988 [ 1590.679101] Key type lgssc registered [ 1591.019377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1593.509554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1594.644856] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1594.657958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1595.954221] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1598.860274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1598.975815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1600.326189] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1602.762766] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1604.908368] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1607.264957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1607.397039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1608.678664] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1610.468894] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 1611.202542] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1619.937399] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1619.937569] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1619.938162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1619.943036] Lustre: Skipped 1 previous similar message [ 1624.654864] Lustre: server umount lustre-OST0000 complete [ 1629.663688] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1629.665687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1629.668918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1629.670738] Lustre: Skipped 1 previous similar message [ 1631.935290] Lustre: server umount lustre-MDT0000 complete [ 1633.136163] LustreError: 120568:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844251 with bad export cookie 1109502926383846523 [ 1633.138635] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1633.139465] LustreError: 120568:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1633.256089] Lustre: server umount lustre-MDT0001 complete [ 1637.900887] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 1638.749512] Key type lgssc unregistered [ 1638.853344] LNet: 124236:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1638.855667] LNet: Removed LNI 192.168.202.117@tcp [ 1639.116528] Key type .llcrypt unregistered [ 1639.118570] Key type ._llcrypt unregistered [ 1647.567956] alg: No test for adler32 (adler32-zlib) [ 1648.319370] Key type ._llcrypt registered [ 1648.320392] Key type .llcrypt registered [ 1648.352578] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 1648.650529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1648.857659] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 1648.931611] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 1648.933149] LNet: Accept secure, port 988 [ 1650.511087] Key type lgssc registered [ 1650.812689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1653.208365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1654.340505] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1654.354115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1655.627970] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1658.224078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1658.340674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1659.614707] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1661.996153] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1663.771427] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1665.966211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1666.113654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1667.909484] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1669.224660] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 1670.932628] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1678.596257] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 1680.108450] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 1680.111480] Lustre: Skipped 2 previous similar messages [ 1681.889525] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 1681.894571] Lustre: Skipped 2 previous similar messages [ 1686.538819] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1686.567185] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1688.925729] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 1688.927842] Lustre: Skipped 2 previous similar messages [ 1692.417487] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.17@tcp 192.168.0.[2,10]@tcp [ 1698.749274] Lustre: server umount lustre-OST0000 complete [ 1704.418181] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1704.418889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1704.430693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1704.434078] Lustre: Skipped 1 previous similar message [ 1705.951231] Lustre: 129644:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759844318/real 1759844318] req@0000000075bb642f x1845330462133824/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1759844324 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1706.047398] Lustre: server umount lustre-MDT0000 complete [ 1706.463101] Lustre: 125007:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759844317/real 1759844317] req@000000006876d1c0 x1845330462132608/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1759844324 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 1706.463287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1706.473909] Lustre: 125007:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1706.479581] Lustre: Skipped 1 previous similar message [ 1706.481765] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1706.489378] LustreError: Skipped 1 previous similar message [ 1707.754613] LustreError: 125677:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844326 with bad export cookie 11718917284404727612 [ 1707.759810] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1707.761371] LustreError: 125677:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1707.905806] Lustre: server umount lustre-MDT0001 complete [ 1713.525900] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 1714.782287] Key type lgssc unregistered [ 1714.948602] LNet: 130447:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1714.953189] LNet: Removed LNI 192.168.202.117@tcp [ 1715.343473] Key type .llcrypt unregistered [ 1715.345030] Key type ._llcrypt unregistered [ 1728.376380] alg: No test for adler32 (adler32-zlib) [ 1729.128312] Key type ._llcrypt registered [ 1729.129427] Key type .llcrypt registered [ 1729.167867] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 1729.569498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1729.694522] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 1729.762112] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 1729.763805] LNet: Accept secure, port 988 [ 1731.343190] Key type lgssc registered [ 1731.917558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1735.078779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1736.236291] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1736.250929] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1736.256084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1736.262887] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.202.17@tcp 192.168.0.[2,10]@tcp [ 1737.672273] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1740.119644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1740.213380] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1740.220230] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.17@tcp 192.168.0.[2,10]@tcp [ 1740.232104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1741.416827] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1743.642776] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1745.371969] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1747.416748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1747.517420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1748.742041] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1751.054394] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1751.657426] Lustre: lustre-OST0000: deleting orphan objects from 0x0:167 to 0x0:193 [ 1760.225387] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1760.225932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1760.228497] Lustre: Skipped 1 previous similar message [ 1760.231266] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1765.344509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1765.346130] Lustre: Skipped 2 previous similar messages [ 1766.030265] Lustre: server umount lustre-OST0000 complete [ 1770.464433] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1770.464797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1770.468241] Lustre: Skipped 1 previous similar message [ 1773.316085] Lustre: server umount lustre-MDT0000 complete [ 1774.437089] LustreError: 132032:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844392 with bad export cookie 11852170650141895953 [ 1774.439252] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1774.439682] LustreError: 132032:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1774.542892] Lustre: server umount lustre-MDT0001 complete [ 1779.166296] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 1780.167938] Key type lgssc unregistered [ 1780.288475] LNet: 135759:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1780.292539] LNet: Removed LNI 192.168.202.117@tcp [ 1780.594532] Key type .llcrypt unregistered [ 1780.595539] Key type ._llcrypt unregistered [ 1789.073502] alg: No test for adler32 (adler32-zlib) [ 1789.826289] Key type ._llcrypt registered [ 1789.827233] Key type .llcrypt registered [ 1789.863168] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 1795.167271] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 1795.640794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1795.831449] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 1795.909299] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 1795.912510] LNet: Accept secure, port 988 [ 1797.519131] Key type lgssc registered [ 1797.909540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1800.954576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1802.889634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1804.745881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1806.771214] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1808.961396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1808.988562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1810.072797] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1810.085204] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1810.125558] Lustre: lustre-MDT0000: new disk, initializing [ 1810.149799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1810.158317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1812.033565] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1817.224842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1817.266914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1817.322615] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1817.343654] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1817.346612] Lustre: Skipped 1 previous similar message [ 1817.394038] Lustre: lustre-MDT0001: new disk, initializing [ 1817.430462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1817.449811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1817.454315] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1819.094937] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1823.750481] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1825.577110] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1828.539953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1828.572166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1828.578297] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1828.700677] Lustre: lustre-OST0000: new disk, initializing [ 1828.702496] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1828.736029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1828.765147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1828.767928] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1830.557841] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1834.269616] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1835.728576] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1835.805654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1837.188171] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1837.265928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1839.074619] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1839.075494] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1839.078149] LustreError: Skipped 1 previous similar message [ 1839.081776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1839.084214] Lustre: Skipped 1 previous similar message [ 1839.956183] Lustre: server umount lustre-OST0000 complete [ 1843.167878] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1843.170493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1843.173472] Lustre: Skipped 1 previous similar message [ 1843.174960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1844.191977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1844.192499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1847.303948] Lustre: server umount lustre-MDT0000 complete [ 1848.429354] LustreError: 140124:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844466 with bad export cookie 6190117848327369474 [ 1848.431793] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1848.432132] LustreError: 140124:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1848.551479] Lustre: server umount lustre-MDT0001 complete [ 1851.144408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1851.301309] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1851.322500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1852.606255] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1855.368294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1855.487105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1857.042902] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1859.353539] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1861.179703] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1863.375069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1863.510227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1864.915529] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1867.240416] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1868.678829] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1868.759856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1870.078490] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1871.163298] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec [ 1874.482385] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1874.512210] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1874.514374] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 1874.567515] Lustre: lustre-OST0001: new disk, initializing [ 1874.569483] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1876.013575] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1877.243921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1877.247295] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1879.317675] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 1879.391803] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1880.791274] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 1880.854985] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1882.611909] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9bfe0589c800.ost_server_uuid 40 [ 1883.069715] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9bfe0589c800.ost_server_uuid in FULL state after 0 sec [ 1887.714273] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1887.714790] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1887.717869] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1887.719119] LustreError: Skipped 1 previous similar message [ 1887.723976] Lustre: Skipped 2 previous similar messages [ 1890.985339] Lustre: server umount lustre-OST0001 complete [ 1892.319973] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1892.322162] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1892.325138] Lustre: Skipped 1 previous similar message [ 1892.326492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1892.832892] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1892.835948] LustreError: Skipped 1 previous similar message [ 1897.953529] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1897.957340] LustreError: Skipped 1 previous similar message [ 1898.339411] Lustre: server umount lustre-OST0000 complete [ 1901.536386] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1901.539115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1901.542214] Lustre: Skipped 1 previous similar message [ 1901.544600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1901.548041] Lustre: Skipped 3 previous similar messages [ 1914.335113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1914.383155] Lustre: server umount lustre-MDT0000 complete [ 1915.594159] LustreError: 147168:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844533 with bad export cookie 6190117848327370314 [ 1915.598502] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1922.798191] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 1923.815113] Key type lgssc unregistered [ 1923.924347] LNet: 150274:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1923.926820] LNet: Removed LNI 192.168.202.117@tcp [ 1924.205803] Key type .llcrypt unregistered [ 1924.207047] Key type ._llcrypt unregistered [ 1934.780905] alg: No test for adler32 (adler32-zlib) [ 1935.532356] Key type ._llcrypt registered [ 1935.533231] Key type .llcrypt registered [ 1935.567602] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 1941.377078] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 1941.766576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1941.915943] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 1941.985991] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 1941.987597] LNet: Accept secure, port 988 [ 1943.559079] Key type lgssc registered [ 1943.857993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1946.531196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1948.512753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1950.429117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1952.425075] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1954.585447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1954.606460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1955.690880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1955.700523] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1955.739783] Lustre: lustre-MDT0000: new disk, initializing [ 1955.764385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1955.770917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1957.091507] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1961.044460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1961.065306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1961.095284] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1961.115537] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1961.117622] Lustre: Skipped 1 previous similar message [ 1961.157456] Lustre: lustre-MDT0001: new disk, initializing [ 1961.193257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1961.204407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1961.208632] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1962.613750] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1966.175268] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1967.870220] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1969.897760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1969.920491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1969.923690] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1970.026874] Lustre: lustre-OST0000: new disk, initializing [ 1970.029120] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1970.054526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1971.480671] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 1971.768835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1971.773591] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1975.189688] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1976.564102] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1976.646996] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1978.082291] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1978.151883] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1981.920551] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1981.922406] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1981.922693] LustreError: Skipped 1 previous similar message [ 1981.926296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1981.928770] Lustre: Skipped 1 previous similar message [ 1987.042378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1987.044180] Lustre: Skipped 1 previous similar message [ 1992.161266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1992.164598] Lustre: Skipped 1 previous similar message [ 1993.183239] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1993.270178] Lustre: server umount lustre-OST0000 complete [ 1997.280431] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1997.280841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1997.281444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1997.287329] Lustre: Skipped 2 previous similar messages [ 2000.712855] Lustre: server umount lustre-MDT0000 complete [ 2002.054052] LustreError: 155213:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844620 with bad export cookie 10387553604703403829 [ 2002.056574] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2002.057809] LustreError: 155213:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2002.191506] Lustre: server umount lustre-MDT0001 complete [ 2005.331269] Lustre: DEBUG MARKER: == conf-sanity test 47: server restart does not make client loss lru_resize settings ========================================================== 09:43:42 (1759844622) [ 2011.923125] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 2015.226706] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 2019.970416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2023.223747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2026.154148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2028.769379] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2031.752088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2031.788268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2031.930189] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2031.949474] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2031.999321] Lustre: lustre-MDT0000: new disk, initializing [ 2032.029944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2032.041242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2034.163389] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2039.015387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2039.056149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2039.105073] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2039.160085] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2039.162759] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2039.164516] Lustre: Skipped 1 previous similar message [ 2039.167983] Lustre: Skipped 1 previous similar message [ 2039.219910] Lustre: lustre-MDT0001: new disk, initializing [ 2039.270700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2039.292946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2039.299707] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2041.287699] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2046.229114] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2048.753167] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2051.732670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2051.770778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2051.776425] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2051.906428] Lustre: lustre-OST0000: new disk, initializing [ 2051.909093] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2051.948202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2054.036324] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2054.417035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2054.422695] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2058.898919] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2061.179911] Lustre: Failing over lustre-OST0000 [ 2061.297623] Lustre: server umount lustre-OST0000 complete [ 2071.903127] Lustre: 152589:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759844683/real 1759844683] req@00000000f58dcefe x1845330769376320/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1759844690 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 2071.909614] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2072.927240] Lustre: 152590:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759844683/real 1759844683] req@00000000b629f3f2 x1845330769375872/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1759844690 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 2072.944873] Lustre: 152590:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2075.463340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2075.655218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2075.666396] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2077.414983] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2077.727325] Lustre: 152590:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759844688/real 1759844688] req@00000000d957dcdf x1845330769376896/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1759844695 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2077.742724] Lustre: 152590:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2078.694654] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2078.833584] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 2078.833940] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 2078.836680] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2079.140839] Lustre: Failing over lustre-MDT0000 [ 2079.319837] Lustre: server umount lustre-MDT0000 complete [ 2080.224465] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2080.229091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2080.235713] Lustre: Skipped 1 previous similar message [ 2080.238544] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2083.809158] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2083.816332] LustreError: Skipped 2 previous similar messages [ 2088.929062] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2088.938591] LustreError: Skipped 3 previous similar messages [ 2090.975168] Lustre: 152591:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759844702/real 1759844702] req@000000009a3eeca8 x1845330769381312/t0(0) o400->MGC192.168.202.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1759844709 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2090.975497] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2090.991387] Lustre: 152591:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2091.000570] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2091.012554] LustreError: Skipped 3 previous similar messages [ 2093.674506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2096.096687] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2096.105028] LustreError: Skipped 3 previous similar messages [ 2097.126168] Lustre: Evicted from MGS (at 192.168.202.117@tcp) after server handle changed from 0x902800f7c5acc27d to 0x902800f7c5acc80a [ 2097.135534] Lustre: MGC192.168.202.117@tcp: Connection restored to (at 0@lo) [ 2097.139956] Lustre: Skipped 1 previous similar message [ 2097.301468] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2099.297537] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2102.380926] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2102.771830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2102.780983] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2102.809628] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 2104.800114] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2104.803633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2104.808915] Lustre: Skipped 2 previous similar messages [ 2104.811267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2109.886839] Lustre: server umount lustre-OST0000 complete [ 2112.993849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2112.993858] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2112.993873] Lustre: Skipped 1 previous similar message [ 2113.003807] LustreError: Skipped 1 previous similar message [ 2118.112353] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2118.117143] LustreError: Skipped 5 previous similar messages [ 2119.414532] LustreError: 161339:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844737 with bad export cookie 10387553604703406090 [ 2119.420400] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2119.427383] LustreError: 161339:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2125.275057] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 2126.263509] Key type lgssc unregistered [ 2126.384427] LNet: 167191:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2126.387417] LNet: Removed LNI 192.168.202.117@tcp [ 2126.679414] Key type .llcrypt unregistered [ 2126.680371] Key type ._llcrypt unregistered [ 2137.725559] alg: No test for adler32 (adler32-zlib) [ 2138.477347] Key type ._llcrypt registered [ 2138.478565] Key type .llcrypt registered [ 2138.540952] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 2139.032547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2139.237663] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 2139.321043] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 2139.322735] LNet: Accept secure, port 988 [ 2140.911098] Key type lgssc registered [ 2141.377475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2144.802339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2145.986972] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2146.011433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2148.194571] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2151.394576] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2152.385038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2152.551664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2154.718792] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2158.176364] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2161.061605] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2164.359612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2164.512791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2166.585620] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2169.943842] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2175.780546] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:97 [ 2300.493497] Lustre: Failing over lustre-MDT0000 [ 2300.607654] Lustre: server umount lustre-MDT0000 complete [ 2300.898186] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2300.902602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2300.909977] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2303.608540] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2303.971332] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2303.980290] Lustre: Skipped 1 previous similar message [ 2308.716854] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2308.725424] LustreError: Skipped 3 previous similar messages [ 2311.135210] Lustre: 167971:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759844922/real 1759844922] req@0000000093f65d4b x1845330976234432/t0(0) o400->MGC192.168.202.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1759844929 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2311.142104] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2314.532511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2317.285829] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2317.285833] Lustre: Evicted from MGS (at 192.168.202.117@tcp) after server handle changed from 0xd6bde88c2222fe1e to 0xd6bde88c22256a38 [ 2317.285856] LustreError: Skipped 11 previous similar messages [ 2317.296405] Lustre: MGC192.168.202.117@tcp: Connection restored to 192.168.202.117@tcp (at 0@lo) [ 2317.414234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2317.450395] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2319.576186] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2321.004065] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2322.923361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2322.936551] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2322.952512] Lustre: lustre-OST0000: deleting orphan objects from 0x0:100 to 0x0:129 [ 2325.804753] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2326.493671] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2343.394707] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2343.395894] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2343.396826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2343.411186] Lustre: Skipped 2 previous similar messages [ 2348.511949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2348.513378] Lustre: Skipped 4 previous similar messages [ 2353.634232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2353.637589] Lustre: Skipped 2 previous similar messages [ 2354.143135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2354.213089] Lustre: server umount lustre-MDT0000 complete [ 2355.813069] LustreError: 169444:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844974 with bad export cookie 15473779583280835128 [ 2355.816991] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2355.821149] LustreError: 169444:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2358.752651] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2358.753070] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2358.767536] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2362.083966] Lustre: server umount lustre-MDT0001 complete [ 2369.862429] Lustre: server umount lustre-OST0000 complete [ 2374.186239] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 2378.258342] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 2382.974397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2385.857656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2388.701454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2391.722949] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2395.060817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2395.112784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2395.265606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2395.283769] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2395.338190] Lustre: lustre-MDT0000: new disk, initializing [ 2395.384543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2395.396184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2397.319914] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2401.968569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2402.011207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2402.059994] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2402.083958] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2402.087569] Lustre: Skipped 1 previous similar message [ 2402.131035] Lustre: lustre-MDT0001: new disk, initializing [ 2402.185691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2402.191567] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2403.727722] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2408.156773] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2410.141833] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2413.463512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2413.509260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2413.514789] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2413.635703] Lustre: lustre-OST0000: new disk, initializing [ 2413.637798] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2413.665179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2413.668905] Lustre: Skipped 1 previous similar message [ 2415.612430] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2419.211839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2419.217740] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2420.296107] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2422.268397] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2422.392369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2424.346059] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2424.470582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2429.412859] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2429.413455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2429.425504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2429.428938] Lustre: Skipped 3 previous similar messages [ 2431.556308] Lustre: server umount lustre-OST0000 complete [ 2439.649645] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2439.654939] LustreError: Skipped 3 previous similar messages [ 2441.378700] LustreError: 175538:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845059 with bad export cookie 15473779583280835849 [ 2441.385688] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2441.388954] LustreError: 175538:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2441.535100] Lustre: server umount lustre-MDT0001 complete [ 2441.536730] Lustre: Skipped 1 previous similar message [ 2445.600458] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:51:03 (1759845063) [ 2453.109560] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 2456.725948] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 2460.979960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2464.198854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2467.315418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2470.210250] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2473.539332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2473.582123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2473.706650] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2473.717036] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2473.763810] Lustre: lustre-MDT0000: new disk, initializing [ 2473.791489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2473.801196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2475.896376] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2481.486095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2481.534346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2481.592847] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2481.623644] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2481.626927] Lustre: Skipped 1 previous similar message [ 2481.671269] Lustre: lustre-MDT0001: new disk, initializing [ 2481.749296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2481.754462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2483.843154] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2488.905190] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2491.686188] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2494.297910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2494.338761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2494.344660] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2494.500846] Lustre: lustre-OST0000: new disk, initializing [ 2494.504846] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2495.954599] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2500.088113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2500.091132] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2500.411720] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2505.185674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2505.186892] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2505.187543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2505.187552] Lustre: Skipped 2 previous similar messages [ 2505.189777] Lustre: Skipped 4 previous similar messages [ 2505.192873] LustreError: Skipped 1 previous similar message [ 2508.988420] Lustre: server umount lustre-OST0000 complete [ 2512.352499] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2518.577372] LustreError: 182515:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845136 with bad export cookie 15473779583280836689 [ 2518.580092] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2518.580977] LustreError: 182515:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2522.411481] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:52:19 (1759845139) [ 2529.643933] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 2533.172238] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 2537.541015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2540.708538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2543.278840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2546.235705] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2549.639890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2549.678370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2549.821645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2549.843383] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2549.904323] Lustre: lustre-MDT0000: new disk, initializing [ 2549.947356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2549.951836] Lustre: Skipped 2 previous similar messages [ 2549.962541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2551.624200] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2557.117335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2557.148866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2557.328583] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2559.536610] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2564.501872] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2566.573830] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2569.669155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2569.706958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2569.710992] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2571.800537] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2573.453941] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2576.467495] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2578.914449] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2578.914987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2578.919629] LustreError: Skipped 1 previous similar message [ 2578.927430] Lustre: Skipped 2 previous similar messages [ 2578.928304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2578.936807] Lustre: Skipped 5 previous similar messages [ 2581.001724] Lustre: server umount lustre-OST0000 complete [ 2581.003739] Lustre: Skipped 2 previous similar messages [ 2597.343188] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2599.268921] LustreError: 188971:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845217 with bad export cookie 15473779583280837753 [ 2599.274338] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2599.275492] LustreError: 188971:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2605.420526] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 2607.027224] Key type lgssc unregistered [ 2607.185601] LNet: 192935:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2607.191783] LNet: Removed LNI 192.168.202.117@tcp [ 2607.641616] Key type .llcrypt unregistered [ 2607.643685] Key type ._llcrypt unregistered [ 2618.656235] alg: No test for adler32 (adler32-zlib) [ 2619.408377] Key type ._llcrypt registered [ 2619.409681] Key type .llcrypt registered [ 2619.450904] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 2619.948412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2620.184283] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 2620.263234] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 2620.265065] LNet: Accept secure, port 988 [ 2621.847161] Key type lgssc registered [ 2622.448088] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2626.525884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2627.738577] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.756845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2629.630453] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2633.063684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2633.224533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2634.980207] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2638.222207] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2640.547774] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2643.321428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2643.435440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2644.079678] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2644.837895] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2647.539119] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2663.904344] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2663.906234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2663.908924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2663.912570] Lustre: Skipped 1 previous similar message [ 2666.304766] Lustre: server umount lustre-OST0000 complete [ 2669.025736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2669.025830] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2669.027244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2669.027252] Lustre: Skipped 1 previous similar message [ 2669.035720] Lustre: Skipped 1 previous similar message [ 2674.144970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2674.150152] Lustre: Skipped 2 previous similar messages [ 2679.264700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2679.268379] Lustre: Skipped 1 previous similar message [ 2682.335172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2682.408708] Lustre: server umount lustre-MDT0000 complete [ 2684.119488] LustreError: 197269:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845302 with bad export cookie 9676405178054545405 [ 2684.123619] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2684.125847] LustreError: 197269:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2684.263430] Lustre: server umount lustre-MDT0001 complete [ 2689.361754] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 2690.283385] Key type lgssc unregistered [ 2690.390335] LNet: 198072:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2690.393622] LNet: Removed LNI 192.168.202.117@tcp [ 2690.703952] Key type .llcrypt unregistered [ 2690.705373] Key type ._llcrypt unregistered [ 2700.788030] alg: No test for adler32 (adler32-zlib) [ 2701.539357] Key type ._llcrypt registered [ 2701.540248] Key type .llcrypt registered [ 2701.578818] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 2701.935991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2702.098433] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 2702.169722] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 2702.171256] LNet: Accept secure, port 988 [ 2703.751156] Key type lgssc registered [ 2704.197227] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2707.502389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2708.697131] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2708.717637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2710.272633] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2713.944055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2714.080120] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2714.135672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2715.498111] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2718.526223] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2720.291557] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2723.139224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2723.302426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2724.326188] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 2724.949416] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2727.747565] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2739.316468] Lustre: lustre-OST0000: Not available for connect from 192.168.202.17@tcp (stopping) [ 2739.681025] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2739.682763] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2739.683834] LustreError: Skipped 1 previous similar message [ 2744.429136] Lustre: lustre-OST0000: Not available for connect from 192.168.202.17@tcp (stopping) [ 2744.433531] Lustre: Skipped 2 previous similar messages [ 2749.407172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2749.479738] Lustre: server umount lustre-OST0000 complete [ 2752.680899] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9bfe11228800.ost_server_uuid 40 [ 2753.406493] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2765.281768] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2765.281853] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2765.282835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2765.282843] Lustre: Skipped 2 previous similar messages [ 2765.290232] Lustre: Skipped 2 previous similar messages [ 2769.888512] Lustre: server umount lustre-MDT0000 complete [ 2771.424952] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2771.433208] LustreError: Skipped 1 previous similar message [ 2771.653596] LustreError: 199512:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845389 with bad export cookie 2186205172641480879 [ 2771.658836] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2771.662682] LustreError: 199512:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2771.821994] Lustre: server umount lustre-MDT0001 complete [ 2775.402592] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 09:56:32 (1759845392) [ 2778.636149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2778.883220] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2778.925841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2780.896975] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2784.224304] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2784.667952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2784.820780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2786.702710] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2789.804386] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2792.502393] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2795.531464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2795.665263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2797.464731] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2800.589942] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2802.772571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2802.796203] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2802.798679] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2802.853409] Lustre: lustre-OST0001: new disk, initializing [ 2802.854930] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2804.320508] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2805.829213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2805.833366] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2808.476742] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2808.935226] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 2810.337560] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2810.341593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2810.346931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2810.348874] Lustre: Skipped 3 previous similar messages [ 2814.438151] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2814.450102] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2815.556766] Lustre: server umount lustre-OST0000 complete [ 2818.173582] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2818.253194] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2819.182984] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.202.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2820.089552] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2820.204172] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2829.420897] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.202.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2829.433270] LustreError: Skipped 5 previous similar messages [ 2831.839881] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2831.839989] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2831.842302] LustreError: Skipped 1 previous similar message [ 2831.846211] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2831.848595] Lustre: Skipped 4 previous similar messages [ 2836.951402] Lustre: server umount lustre-OST0001 complete [ 2840.032590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2840.035796] Lustre: Skipped 2 previous similar messages [ 2845.646024] LustreError: 203481:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845463 with bad export cookie 2186205172641481929 [ 2845.648351] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2845.648673] LustreError: 203481:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2845.773313] Lustre: server umount lustre-MDT0001 complete [ 2845.774536] Lustre: Skipped 1 previous similar message [ 2847.677259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2849.001451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2850.224487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2851.512166] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2853.564099] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 09:57:51 (1759845471) [ 2855.783778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2855.884681] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2855.897296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2855.966110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2855.967932] Lustre: Skipped 1 previous similar message [ 2857.778670] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2863.861823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2863.958640] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2863.986757] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2867.044723] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2871.871217] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2875.634500] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2880.036660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2880.186179] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2883.888450] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2887.828192] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:129 [ 2894.075535] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2896.434213] hrtimer: interrupt took 2557608 ns [ 2902.010533] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2902.285872] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2902.598867] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2902.612355] Lustre: Skipped 2 previous similar messages [ 2908.080262] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 2916.844322] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2920.931162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2920.956178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2920.958454] Lustre: Skipped 4 previous similar messages [ 2925.275963] Lustre: server umount lustre-OST0000 complete [ 2926.050089] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2926.064555] LustreError: Skipped 7 previous similar messages [ 2941.409826] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2954.351701] LustreError: 209889:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845572 with bad export cookie 2186205172641483308 [ 2954.355213] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2954.356491] LustreError: 209889:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2957.289122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2959.434915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2961.453513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2963.265076] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2966.417063] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 09:59:43 (1759845583) [ 2975.128955] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 2980.552440] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 2986.182205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2990.041446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2993.918295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2997.514456] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3001.047744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3001.078716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3001.210134] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3001.228898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3001.276889] Lustre: lustre-MDT0000: new disk, initializing [ 3001.322848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3001.336915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3003.541373] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3009.204979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3009.239290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3009.297384] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3009.317943] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3009.320224] Lustre: Skipped 1 previous similar message [ 3009.360666] Lustre: lustre-MDT0001: new disk, initializing [ 3009.430397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3009.436366] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3011.309038] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3016.021360] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3018.804446] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3021.859610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3021.896435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3021.900263] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3022.050848] Lustre: lustre-OST0000: new disk, initializing [ 3022.057076] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3023.839832] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3028.271728] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3028.985670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3028.989802] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3029.926326] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3030.008602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3031.844061] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3031.939410] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3034.080618] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3034.081686] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3034.083712] LustreError: Skipped 1 previous similar message [ 3034.086913] Lustre: Skipped 5 previous similar messages [ 3034.088875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3034.091145] Lustre: Skipped 7 previous similar messages [ 3038.861346] Lustre: server umount lustre-OST0000 complete [ 3038.863356] Lustre: Skipped 3 previous similar messages [ 3049.218479] LustreError: 218097:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845667 with bad export cookie 2186205172641484827 [ 3049.220809] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3049.226018] LustreError: 218097:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3054.259880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3054.527734] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3054.533732] LustreError: Skipped 8 previous similar messages [ 3056.803943] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3061.147414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3063.528219] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3067.462849] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3070.404411] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3073.654604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3075.832303] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3079.139047] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3081.196869] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3084.459799] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3086.469892] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3086.584127] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3089.379486] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3089.384065] LustreError: Skipped 1 previous similar message [ 3096.442438] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3098.820434] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3124.699816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3127.274741] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3132.005724] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3133.158452] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3135.122709] Lustre: server umount lustre-OST0000 complete [ 3135.124089] Lustre: Skipped 3 previous similar messages [ 3137.504424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3137.507445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3137.516849] Lustre: Skipped 5 previous similar messages [ 3137.522452] Lustre: Skipped 5 previous similar messages [ 3145.682310] LustreError: 225765:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845763 with bad export cookie 2186205172641485660 [ 3145.683874] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3145.688468] LustreError: 225765:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3150.347189] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:02:47 (1759845767) [ 3153.920337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3154.138904] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3154.145621] LustreError: Skipped 1 previous similar message [ 3154.178562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3154.181162] Lustre: Skipped 6 previous similar messages [ 3156.293468] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3160.939171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3163.770817] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3168.741477] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3171.928586] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3175.497801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3177.928193] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3181.369055] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3183.303993] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3184.463453] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3186.440988] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3186.534202] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3189.783397] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3189.832320] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3189.835500] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3189.897161] Lustre: lustre-OST0001: new disk, initializing [ 3189.899989] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3191.765151] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3196.494449] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3198.391577] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3198.484840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3198.489952] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3199.516212] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3201.149618] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 3201.230934] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3203.553687] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3203.556246] LustreError: Skipped 1 previous similar message [ 3210.070277] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3210.143322] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3211.538426] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 3211.616101] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3235.088815] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3236.485970] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3238.963676] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3239.514847] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3264.281058] Lustre: server umount lustre-OST0000 complete [ 3264.282341] Lustre: Skipped 4 previous similar messages [ 3268.575869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3268.579622] Lustre: Skipped 4 previous similar messages [ 3268.581444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3268.583349] Lustre: Skipped 12 previous similar messages [ 3273.044837] LustreError: 235598:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845891 with bad export cookie 2186205172641486598 [ 3273.047721] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3273.048047] LustreError: 235598:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3281.171315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3282.445446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3283.730154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3284.961034] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3287.224714] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:05:04 (1759845904) [ 3289.631293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3289.746145] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3289.755743] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3291.424139] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3294.259730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3294.289383] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3295.844267] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3298.617566] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3300.879144] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3303.268677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3303.343359] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3304.932355] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3307.828525] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3317.920032] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3317.955475] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3319.886848] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3322.651131] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3327.501883] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 3328.930433] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3329.004843] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3330.380084] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 3330.448565] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3332.340296] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9bfe08c5a000.ost_server_uuid 40 [ 3332.812601] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9bfe08c5a000.ost_server_uuid in FULL state after 0 sec [ 3333.121284] Lustre: Permanently deactivating lustre-OST0001 [ 3333.125435] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 3333.128690] Lustre: Skipped 1 previous similar message [ 3333.946498] Lustre: Permanently reactivating lustre-OST0001 [ 3336.675386] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3336.678294] LustreError: Skipped 2 previous similar messages [ 3341.795522] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3341.801066] LustreError: Skipped 26 previous similar messages [ 3364.319125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3365.646054] LustreError: 237582:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845983 with bad export cookie 2186205172641487851 [ 3365.648886] LustreError: 237582:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3365.649023] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3367.600894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3369.059255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3370.545872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3372.161945] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3374.433126] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:06:32 (1759845992) [ 3374.789486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3377.403088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3377.536058] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3377.552413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3377.554721] Lustre: Skipped 5 previous similar messages [ 3379.224823] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3382.431543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3382.463148] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3383.965329] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3387.126979] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3389.187273] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3391.726639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3391.748847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3391.824337] Lustre: Permanently deactivating lustre-OST0000 [ 3393.362728] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3394.875518] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3396.606835] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3399.481695] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3401.420689] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3404.334399] Lustre: lustre-OST0001: deleting orphan objects from 0x0:35 to 0x0:65 [ 3404.689117] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3405.465152] Lustre: Permanently reactivating lustre-OST0000 [ 3412.978784] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3412.980671] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3412.983362] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 3412.996244] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3413.004658] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 3437.023182] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3446.209914] LustreError: 244423:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759846064 with bad export cookie 2186205172641489440 [ 3446.212217] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3446.214212] LustreError: 244423:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3450.578195] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 3451.459568] Key type lgssc unregistered [ 3451.556346] LNet: 249368:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3451.558620] LNet: Removed LNI 192.168.202.117@tcp [ 3451.831562] Key type .llcrypt unregistered [ 3451.832865] Key type ._llcrypt unregistered [ 3461.629128] alg: No test for adler32 (adler32-zlib) [ 3462.381332] Key type ._llcrypt registered [ 3462.382221] Key type .llcrypt registered [ 3462.427433] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 3462.906836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3463.083323] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 3463.160518] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 3463.161805] LNet: Accept secure, port 988 [ 3464.743099] Key type lgssc registered [ 3465.326586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3466.663629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3469.029310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3470.247255] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3470.281217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3472.137464] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3475.426207] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3475.599033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3475.632397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3475.686104] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3475.703325] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3475.707490] Lustre: Permanently deactivating lustre-MDT0001 [ 3475.892295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3476.973583] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3476.975472] LustreError: 250913:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3476.980938] LustreError: 250913:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3476.987356] LustreError: 250913:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -108 [ 3477.799455] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3481.407164] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3483.703095] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3486.108148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3486.274337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3488.124675] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3491.426923] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3491.434388] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:131 to 0x280000400:161 [ 3491.640560] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3494.377666] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3494.505654] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3496.251654] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3499.614743] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3500.007487] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:67 to 0x2c0000400:97 [ 3501.040873] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 3501.041345] Lustre: lustre-OST0001: deleting orphan objects from 0x0:35 to 0x0:97 [ 3505.863414] Lustre: Permanently reactivating lustre-MDT0001 [ 3505.873980] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 3505.877616] Lustre: Skipped 3 previous similar messages [ 3506.354082] Lustre: lustre-MDT0001: Client 95755cc3-7eb1-46ff-8c38-70a3a6db600e (at 192.168.202.17@tcp) reconnecting [ 3509.528241] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3510.330068] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3512.789842] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 3512.907878] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3514.867937] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 3514.971289] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3516.115207] Lustre: Permanently deactivating lustre-MDT0001 [ 3516.126867] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 3516.129665] Lustre: Skipped 2 previous similar messages [ 3518.485646] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3531.749921] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3531.752447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3531.757477] Lustre: Skipped 3 previous similar messages [ 3531.761030] Lustre: Skipped 2 previous similar messages [ 3535.845177] Lustre: server umount lustre-MDT0000 complete [ 3536.864663] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3536.868828] LustreError: Skipped 3 previous similar messages [ 3537.349657] LustreError: 250862:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759846155 with bad export cookie 8582509596833085365 [ 3537.355695] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3537.504941] Lustre: server umount lustre-MDT0001 complete [ 3545.134711] Lustre: server umount lustre-OST0000 complete [ 3549.994137] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:09:27 (1759846167) [ 3555.097944] Lustre: server umount lustre-OST0001 complete [ 3558.407202] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 3562.054984] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 3566.302530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3569.135884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3571.717808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3574.420911] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3577.037678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3577.062398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3577.137715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3577.140377] Lustre: Skipped 2 previous similar messages [ 3577.151463] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3577.191205] Lustre: lustre-MDT0000: new disk, initializing [ 3577.214482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3577.222448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3578.942986] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3584.516463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3584.542837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3584.583115] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3584.608478] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3584.610662] Lustre: Skipped 1 previous similar message [ 3584.651192] Lustre: lustre-MDT0001: new disk, initializing [ 3584.690736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3584.693067] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3586.486844] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3591.582551] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3594.414363] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3597.429905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3597.466082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3597.473710] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3597.606416] Lustre: lustre-OST0000: new disk, initializing [ 3597.609608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3597.653366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3597.657112] Lustre: Skipped 1 previous similar message [ 3599.521680] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3601.217677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3601.223105] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3604.064743] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3604.873703] LustreError: 260765:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3609.652127] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3609.679269] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3609.683593] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3609.744814] Lustre: lustre-OST0001: new disk, initializing [ 3609.747669] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3611.604572] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3614.968110] LustreError: 260765:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 3614.979447] LustreError: 260765:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3616.373812] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3616.770393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3616.773929] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3617.095232] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3625.071138] LustreError: 260765:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 3626.977330] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3626.981689] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3626.985906] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3626.988209] Lustre: Skipped 1 previous similar message [ 3628.664375] Lustre: lustre-OST0001: Not available for connect from 192.168.202.17@tcp (stopping) [ 3628.667847] Lustre: Skipped 1 previous similar message [ 3632.100312] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3632.104645] Lustre: Skipped 1 previous similar message [ 3637.221522] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3637.225098] Lustre: Skipped 2 previous similar messages [ 3640.287151] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3640.312836] Lustre: server umount lustre-OST0001 complete [ 3642.341581] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3642.348590] LustreError: Skipped 1 previous similar message [ 3652.579442] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3652.585019] LustreError: Skipped 3 previous similar messages [ 3657.700600] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3657.701086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3657.703325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3657.703333] Lustre: Skipped 1 previous similar message [ 3657.706256] LustreError: Skipped 2 previous similar messages [ 3657.715531] Lustre: Skipped 1 previous similar message [ 3659.059808] Lustre: server umount lustre-OST0000 complete [ 3661.791973] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3661.796298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3668.297566] LustreError: 259366:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759846286 with bad export cookie 8582509596833088151 [ 3668.303918] LustreError: 259366:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3668.303967] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3679.790449] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 3680.890829] Key type lgssc unregistered [ 3681.008345] LNet: 264571:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3681.010582] LNet: Removed LNI 192.168.202.117@tcp [ 3681.294323] Key type .llcrypt unregistered [ 3681.295292] Key type ._llcrypt unregistered [ 3695.012209] alg: No test for adler32 (adler32-zlib) [ 3695.763381] Key type ._llcrypt registered [ 3695.764739] Key type .llcrypt registered [ 3695.799166] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 3702.859077] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 3703.303000] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3703.423488] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 3703.491190] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 3703.492719] LNet: Accept secure, port 988 [ 3705.071125] Key type lgssc registered [ 3705.526260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3709.587253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3712.333947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3715.208217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3717.546990] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3720.665095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3720.701535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3721.804103] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3721.818417] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3721.859493] Lustre: lustre-MDT0000: new disk, initializing [ 3721.884507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3721.893390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3723.751374] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3728.872319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3728.911131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3728.956797] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3728.983919] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3728.987313] Lustre: Skipped 1 previous similar message [ 3729.029655] Lustre: lustre-MDT0001: new disk, initializing [ 3729.074207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3729.093107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3729.097777] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3730.875515] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3735.769977] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3738.465726] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3741.588928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3741.610604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3741.612675] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3741.692737] Lustre: lustre-OST0000: new disk, initializing [ 3741.694197] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3741.717571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3743.574995] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3748.430730] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3750.300093] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3751.943167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3751.948656] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3752.502391] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3754.416405] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3754.542807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3757.027774] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3757.027861] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3757.032294] LustreError: Skipped 1 previous similar message [ 3757.041195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3757.047464] Lustre: Skipped 1 previous similar message [ 3761.592440] Lustre: server umount lustre-OST0000 complete [ 3765.216281] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3765.221676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3765.229668] Lustre: Skipped 1 previous similar message [ 3765.233183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3767.264741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3767.265463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3769.436374] Lustre: server umount lustre-MDT0000 complete [ 3771.152103] LustreError: 270756:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759846389 with bad export cookie 9865012229860820599 [ 3771.157728] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3771.161165] LustreError: 270756:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3771.289260] Lustre: server umount lustre-MDT0001 complete [ 3774.659420] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 10:13:12 (1759846392) [ 3777.555076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3777.781821] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3777.814377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3779.205121] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3782.594041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3782.778698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3784.522193] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3787.134328] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3789.238925] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3791.488965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3791.631332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3793.089899] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3795.452961] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3805.152321] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3805.154705] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3805.158433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3805.159983] Lustre: Skipped 1 previous similar message [ 3808.225481] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3810.581564] Lustre: server umount lustre-OST0000 complete [ 3811.947520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3816.061602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3816.227707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3817.709974] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3821.042994] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3824.419822] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3824.425060] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.117@tcp (at 0@lo) [ 3824.427526] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 3829.538935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3829.542821] Lustre: Skipped 2 previous similar messages [ 3829.546330] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3829.552941] Lustre: Skipped 1 previous similar message [ 3830.861181] Lustre: server umount lustre-OST0000 complete [ 3833.824341] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3839.968643] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3839.976322] LustreError: Skipped 1 previous similar message [ 3840.014537] LustreError: 277539:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759846458 with bad export cookie 9865012229860821439 [ 3840.020066] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3840.020902] LustreError: 277539:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3840.152483] Lustre: server umount lustre-MDT0001 complete [ 3840.153609] Lustre: Skipped 1 previous similar message [ 3845.478532] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 3846.559081] Key type lgssc unregistered [ 3846.694614] LNet: 278636:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3846.698682] LNet: Removed LNI 192.168.202.117@tcp [ 3847.106217] Key type .llcrypt unregistered [ 3847.107108] Key type ._llcrypt unregistered [ 3857.678403] alg: No test for adler32 (adler32-zlib) [ 3858.431334] Key type ._llcrypt registered [ 3858.432192] Key type .llcrypt registered [ 3858.476761] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 3858.823678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3859.010492] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 3859.079678] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 3859.081113] LNet: Accept secure, port 988 [ 3860.655131] Key type lgssc registered [ 3861.229125] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3864.840950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3865.985117] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3866.001609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3867.872890] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3871.201159] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3871.742419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3871.928682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3873.694738] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3877.287682] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3879.969692] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3883.085213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3883.230292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3885.191714] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3888.741737] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3896.553782] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 3911.345871] Lustre: server umount lustre-OST0000 complete [ 3913.184361] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3913.188761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3913.195983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3917.280794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3917.281444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3919.327348] Lustre: 283751:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759846531/real 1759846531] req@0000000090082e60 x1845332779479616/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1759846537 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3919.327520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3919.340445] Lustre: 283751:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3919.350117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3919.356014] Lustre: Skipped 2 previous similar messages [ 3919.439365] Lustre: server umount lustre-MDT0000 complete [ 3920.351198] Lustre: 279409:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759846531/real 1759846531] req@000000008803e658 x1845332779479680/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1759846538 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 3921.106987] LustreError: 282057:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759846539 with bad export cookie 3582996888410935547 [ 3921.109798] LustreError: 282057:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3921.110180] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3921.232546] Lustre: server umount lustre-MDT0001 complete [ 3926.339427] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 3927.509377] Key type lgssc unregistered [ 3927.630479] LNet: 284554:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3927.633817] LNet: Removed LNI 192.168.202.117@tcp [ 3928.041046] Key type .llcrypt unregistered [ 3928.042720] Key type ._llcrypt unregistered [ 3935.661258] alg: No test for adler32 (adler32-zlib) [ 3936.413300] Key type ._llcrypt registered [ 3936.414559] Key type .llcrypt registered [ 3936.458893] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 3936.796388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3936.991061] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 3937.068398] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 3937.069838] LNet: Accept secure, port 988 [ 3938.647150] Key type lgssc registered [ 3939.115268] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3942.642441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3943.820135] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3943.837389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3945.725968] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3949.024754] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3949.715431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3949.899893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3951.650116] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3955.022562] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3957.636394] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3960.700773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3960.879286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3962.988391] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 3966.467830] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3974.185306] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 3982.818562] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3982.823794] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3982.832732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3984.356319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3984.356588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3987.128445] Lustre: server umount lustre-OST0000 complete [ 3989.473925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3989.474642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3989.480666] Lustre: Skipped 1 previous similar message [ 3989.484743] Lustre: Skipped 1 previous similar message [ 3994.591992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3994.594622] Lustre: Skipped 2 previous similar messages [ 3999.712302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4003.295157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4003.366263] Lustre: server umount lustre-MDT0000 complete [ 4005.189095] LustreError: 288660:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759846623 with bad export cookie 12218605223262868729 [ 4005.192638] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4005.193915] LustreError: 288660:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4005.368431] Lustre: server umount lustre-MDT0001 complete [ 4011.332147] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 4012.562132] Key type lgssc unregistered [ 4012.794343] LNet: 289668:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4012.797907] LNet: Removed LNI 192.168.202.117@tcp [ 4013.136269] Key type .llcrypt unregistered [ 4013.137959] Key type ._llcrypt unregistered [ 4021.774280] alg: No test for adler32 (adler32-zlib) [ 4022.526469] Key type ._llcrypt registered [ 4022.528045] Key type .llcrypt registered [ 4022.574921] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 4023.054573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4023.275228] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 4023.383530] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 4023.385911] LNet: Accept secure, port 988 [ 4024.999184] Key type lgssc registered [ 4026.624297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4042.289032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4044.448138] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4044.568736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4049.889596] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4053.039391] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 4055.010311] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4060.128979] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4064.378571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4064.918475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4069.843704] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 4077.745060] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4083.379669] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4090.821661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4091.083170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4094.563152] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 4095.219076] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 4104.499716] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4115.940866] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4115.945348] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4115.960430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4118.969394] Lustre: server umount lustre-OST0000 complete [ 4126.177044] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4126.177224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4126.189694] Lustre: Skipped 1 previous similar message [ 4126.193236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4126.199175] Lustre: Skipped 1 previous similar message [ 4128.068486] Lustre: server umount lustre-MDT0000 complete [ 4131.110399] LustreError: 290916:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759846749 with bad export cookie 16374071380279269026 [ 4131.114140] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4131.121636] LustreError: 290916:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4131.361046] Lustre: server umount lustre-MDT0001 complete [ 4140.826320] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 4143.097721] Key type lgssc unregistered [ 4143.353782] LNet: 294594:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4143.363475] LNet: Removed LNI 192.168.202.117@tcp [ 4144.045904] Key type .llcrypt unregistered [ 4144.052578] Key type ._llcrypt unregistered [ 4161.792247] alg: No test for adler32 (adler32-zlib) [ 4162.557444] Key type ._llcrypt registered [ 4162.561441] Key type .llcrypt registered [ 4162.646536] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 4163.277497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4163.810858] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 4163.979679] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 4163.983359] LNet: Accept secure, port 988 [ 4165.600373] Key type lgssc registered [ 4166.376675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4172.678521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4174.044750] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4174.096520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4177.585954] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 4179.435259] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4184.546248] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4184.573149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4184.796204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4188.021790] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 4194.419417] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4199.096844] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4204.244971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4204.484878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4208.134968] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 4213.753298] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4223.923887] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 4244.449574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4244.457105] Lustre: Skipped 1 previous similar message [ 4244.459449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4244.461698] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4244.469575] Lustre: Skipped 1 previous similar message [ 4247.876263] Lustre: server umount lustre-OST0000 complete [ 4251.617386] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4251.622149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4251.629660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4254.693164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4254.694387] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4257.320352] Lustre: server umount lustre-MDT0000 complete [ 4259.556597] LustreError: 296843:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759846877 with bad export cookie 9589019713501952187 [ 4259.561942] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4259.566105] LustreError: 296843:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4259.772964] Lustre: server umount lustre-MDT0001 complete [ 4269.215528] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 4271.720358] Key type lgssc unregistered [ 4271.971079] LNet: 300561:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4271.982932] LNet: Removed LNI 192.168.202.117@tcp [ 4272.831923] Key type .llcrypt unregistered [ 4272.833814] Key type ._llcrypt unregistered [ 4285.198950] alg: No test for adler32 (adler32-zlib) [ 4285.974941] Key type ._llcrypt registered [ 4285.976292] Key type .llcrypt registered [ 4286.077491] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 4286.861250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4287.482966] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 4287.714162] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 4287.721601] LNet: Accept secure, port 988 [ 4289.407307] Key type lgssc registered [ 4290.402477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4297.641238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4299.173668] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4299.243649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4303.886641] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 4304.352453] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4309.475188] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4311.924127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4312.317403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4315.982834] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 4322.987891] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4328.124752] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4333.644448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4333.947417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4337.580652] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 4343.909203] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4349.306967] Lustre: lustre-OST0000: deleting orphan objects from 0x0:163 to 0x0:193 [ 4357.600658] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4357.615622] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4357.630882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4359.650050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4359.650693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4359.651931] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4359.664108] Lustre: Skipped 1 previous similar message [ 4363.494445] Lustre: server umount lustre-OST0000 complete [ 4368.875662] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4368.879622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4368.902459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4372.938670] Lustre: server umount lustre-MDT0000 complete [ 4375.008343] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4375.017456] LustreError: Skipped 1 previous similar message [ 4376.350194] LustreError: 301809:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759846994 with bad export cookie 4921073834188171975 [ 4376.358481] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4376.374902] LustreError: 301809:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4376.806974] Lustre: server umount lustre-MDT0001 complete [ 4387.081913] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 4389.620618] Key type lgssc unregistered [ 4389.898731] LNet: 305665:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4389.925562] LNet: Removed LNI 192.168.202.117@tcp [ 4390.575058] Key type .llcrypt unregistered [ 4390.582656] Key type ._llcrypt unregistered [ 4402.996123] alg: No test for adler32 (adler32-zlib) [ 4403.752453] Key type ._llcrypt registered [ 4403.753994] Key type .llcrypt registered [ 4403.881409] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 4404.589595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4405.300177] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 4405.437299] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 4405.441481] LNet: Accept secure, port 988 [ 4407.119234] Key type lgssc registered [ 4408.067723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4414.745220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4416.166814] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4416.230213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4419.680298] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 4421.601612] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4425.876329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4426.162608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4429.551325] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 4434.611795] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4438.508442] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4442.922729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4443.230291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4446.315695] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 4446.870685] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 4452.722760] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4462.050552] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4462.052867] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4462.056780] LustreError: Skipped 1 previous similar message [ 4462.057886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4462.068386] Lustre: Skipped 1 previous similar message [ 4467.176012] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4467.185733] Lustre: Skipped 2 previous similar messages [ 4467.956978] Lustre: server umount lustre-OST0000 complete [ 4472.288250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4472.289710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4472.293940] Lustre: Skipped 1 previous similar message [ 4476.785170] Lustre: server umount lustre-MDT0000 complete [ 4477.409626] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4477.416467] LustreError: Skipped 1 previous similar message [ 4478.938911] LustreError: 306914:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759847097 with bad export cookie 5190462783642862511 [ 4478.943994] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4479.111327] Lustre: server umount lustre-MDT0001 complete [ 4487.002214] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 4488.882516] Key type lgssc unregistered [ 4489.090535] LNet: 310585:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4489.099225] LNet: Removed LNI 192.168.202.117@tcp [ 4489.675487] Key type .llcrypt unregistered [ 4489.677656] Key type ._llcrypt unregistered [ 4499.798081] alg: No test for adler32 (adler32-zlib) [ 4500.565336] Key type ._llcrypt registered [ 4500.567721] Key type .llcrypt registered [ 4500.663894] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4514.577418] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 4524.701509] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 4525.574987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4525.956542] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 4526.179444] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 4526.188351] LNet: Accept secure, port 988 [ 4527.833057] Key type lgssc registered [ 4528.743941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4535.452115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4540.501354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4545.549270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4550.612988] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4556.203841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4556.292308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4557.645500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4557.675571] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4557.748450] Lustre: lustre-MDT0000: new disk, initializing [ 4557.885673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4557.910640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4561.969860] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 4570.960785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4571.020505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4571.089960] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4571.138927] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4571.142374] Lustre: Skipped 1 previous similar message [ 4571.228610] Lustre: lustre-MDT0001: new disk, initializing [ 4571.306492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4571.371571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4571.375715] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4575.337618] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 4583.746604] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4588.709429] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4594.144943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4594.224218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4594.230331] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4594.389958] Lustre: lustre-OST0000: new disk, initializing [ 4594.392888] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4594.462916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4598.059621] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 4599.850252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4599.855483] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4605.971610] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4610.023652] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4610.208899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4614.100134] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4614.275369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4620.256088] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4620.269565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4620.278204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4620.768814] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4620.776057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4620.797271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4621.959996] Lustre: server umount lustre-OST0000 complete [ 4625.889463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4625.897846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4631.008228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4631.011248] Lustre: Skipped 2 previous similar messages [ 4631.246154] Lustre: server umount lustre-MDT0000 complete [ 4634.177276] LustreError: 314328:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759847252 with bad export cookie 2957352545627435286 [ 4634.187453] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4634.187667] LustreError: 314328:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4634.473090] Lustre: server umount lustre-MDT0001 complete [ 4642.036928] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 10:27:38 (1759847258) [ 4648.317761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4648.666462] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4648.753844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4652.299210] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 4654.048833] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4658.679466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4658.962356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4662.201292] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 4667.481802] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4671.101750] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4675.061417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4675.306515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4677.834693] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 4683.156714] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4695.008940] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4695.009033] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4695.020629] LustreError: Skipped 1 previous similar message [ 4695.031279] Lustre: Skipped 1 previous similar message [ 4695.034939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4695.242814] Lustre: server umount lustre-OST0000 complete [ 4700.134139] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4700.135330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4700.148021] Lustre: Skipped 1 previous similar message [ 4704.132646] Lustre: server umount lustre-MDT0000 complete [ 4705.249661] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4705.255781] LustreError: Skipped 1 previous similar message [ 4706.591201] LustreError: 318864:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759847324 with bad export cookie 2957352545627436133 [ 4706.593481] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4706.601530] LustreError: 318864:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4713.153312] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 4715.008810] Key type lgssc unregistered [ 4715.191691] LNet: 322537:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4715.196967] LNet: Removed LNI 192.168.202.117@tcp [ 4715.730555] Key type .llcrypt unregistered [ 4715.732686] Key type ._llcrypt unregistered [ 4735.738562] alg: No test for adler32 (adler32-zlib) [ 4736.495418] Key type ._llcrypt registered [ 4736.498050] Key type .llcrypt registered [ 4736.600579] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 4746.478333] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 4747.018614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4747.413328] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 4747.549747] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 4747.553282] LNet: Accept secure, port 988 [ 4749.199122] Key type lgssc registered [ 4750.140095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4756.282098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4760.559917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4764.504931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4769.429160] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4773.529194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4777.704875] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4782.773246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4782.822927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4783.988256] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4784.025363] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4784.093199] Lustre: lustre-MDT0000: new disk, initializing [ 4784.152964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4784.171142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4786.997181] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 4794.536594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4794.590798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4794.655603] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4794.728818] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4794.733294] Lustre: Skipped 1 previous similar message [ 4794.797218] Lustre: lustre-MDT0001: new disk, initializing [ 4794.886124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4794.923304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4794.932098] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4798.103695] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 4805.207334] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4809.698553] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4814.474794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4814.522687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4814.527858] Lustre: lustre-OST2710-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4814.670778] Lustre: lustre-OST2710: new disk, initializing [ 4814.675409] Lustre: srv-lustre-OST2710: No data found on store. Initialize space: rc = -61 [ 4814.729049] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4818.040545] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 4823.578601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4823.584905] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4824.609759] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4829.113634] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4829.167524] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4829.175742] Lustre: lustre-OST03e8-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4829.261262] Lustre: lustre-OST03e8: new disk, initializing [ 4829.263789] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space: rc = -61 [ 4829.319463] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4832.221892] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 4836.968531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4836.973269] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4837.412923] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4842.683751] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 40 [ 4842.823633] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4845.798135] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 40 [ 4845.946969] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4849.160823] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 40 [ 4849.363899] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4852.745929] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 40 [ 4852.974916] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4856.287990] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4856.292945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4856.305431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4857.830263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4857.831495] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4861.816661] Lustre: server umount lustre-MDT0000 complete [ 4862.948906] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4862.964667] LustreError: Skipped 3 previous similar messages [ 4864.221954] LustreError: 326478:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759847482 with bad export cookie 14883117297360410563 [ 4864.224064] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4864.232449] LustreError: 326478:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4864.466628] Lustre: server umount lustre-MDT0001 complete [ 4873.695154] Lustre: 331969:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759847485/real 1759847485] req@0000000014e46f9e x1845333710631104/t0(0) o39->lustre-MDT0001-lwp-OST2710@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759847491 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4873.764350] Lustre: server umount lustre-OST2710 complete [ 4876.191410] Lustre: 324281:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759847487/real 1759847487] req@00000000568aca4d x1845333710631296/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759847494 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4876.199024] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4876.218020] Lustre: 324281:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4876.228867] Lustre: Skipped 2 previous similar messages [ 4876.403784] Lustre: server umount lustre-OST03e8 complete [ 4890.299144] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 4897.306824] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 4904.092297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4908.600331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4912.956657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4917.391044] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4921.845686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4921.898946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4922.070939] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4922.098497] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4922.190857] Lustre: lustre-MDT0000: new disk, initializing [ 4922.280146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4922.300702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4925.324777] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 4933.352543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4933.397098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4933.476523] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4933.521125] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4933.524326] Lustre: Skipped 1 previous similar message [ 4933.572732] Lustre: lustre-MDT0001: new disk, initializing [ 4933.654101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4933.677079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4933.686591] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4936.811165] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 4943.428275] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4946.899679] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4951.407898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4951.464051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4951.468190] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4951.632116] Lustre: lustre-OST0000: new disk, initializing [ 4951.635962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4951.701874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4955.100658] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 4958.768538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4958.773751] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4962.149795] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4965.214206] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4965.399746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4969.025567] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4969.201777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4974.565113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4974.566340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4974.567398] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4974.569758] Lustre: Skipped 1 previous similar message [ 4974.571898] Lustre: Skipped 3 previous similar messages [ 4979.681879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4979.690400] Lustre: Skipped 2 previous similar messages [ 4984.806238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4984.814019] Lustre: Skipped 1 previous similar message [ 4985.314896] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4985.462693] Lustre: server umount lustre-OST0000 complete [ 4989.923279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4989.923956] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4989.930951] Lustre: Skipped 1 previous similar message [ 4989.939280] LustreError: Skipped 1 previous similar message [ 4994.362214] Lustre: server umount lustre-MDT0000 complete [ 4995.041439] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4995.052886] LustreError: Skipped 1 previous similar message [ 4997.073843] LustreError: 335941:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759847615 with bad export cookie 14883117297360412180 [ 4997.077738] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4997.083890] LustreError: 335941:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5002.840235] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 10:33:40 (1759847620) [ 5003.939345] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 5004.978064] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 10:33:42 (1759847622) [ 5006.998421] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 5011.912225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5014.616459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5017.579278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5020.861930] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5021.737960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5026.520048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5026.739993] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5026.757965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5026.863073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5029.832723] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5035.931155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5036.002586] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5036.029064] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5039.528960] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5044.686011] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5048.435711] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5052.629688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5052.672214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5052.796150] Lustre: Denying initial registration attempt from nid 192.168.202.117@tcp, specified as failover [ 5052.801215] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 5052.805864] LustreError: 343096:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -99 [ 5052.810137] LustreError: 343096:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 5052.813179] LustreError: 343096:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 5052.897842] Lustre: server umount lustre-OST0000 complete [ 5052.899714] Lustre: Skipped 1 previous similar message [ 5052.901186] LustreError: 343096:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 5056.992685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5056.992695] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5057.005486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5057.009454] Lustre: Skipped 3 previous similar messages [ 5062.115244] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5062.126287] LustreError: Skipped 1 previous similar message [ 5063.912852] LustreError: 341140:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759847682 with bad export cookie 14883117297360413034 [ 5063.917847] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5063.922029] LustreError: 341140:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5072.034580] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 5073.518909] Key type lgssc unregistered [ 5073.700852] LNet: 344171:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5073.706830] LNet: Removed LNI 192.168.202.117@tcp [ 5074.125332] Key type .llcrypt unregistered [ 5074.128144] Key type ._llcrypt unregistered [ 5081.186710] alg: No test for adler32 (adler32-zlib) [ 5081.938444] Key type ._llcrypt registered [ 5081.939824] Key type .llcrypt registered [ 5082.002395] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 5082.631045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5082.812913] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 5082.935388] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 5082.938561] LNet: Accept secure, port 988 [ 5084.543139] Key type lgssc registered [ 5085.074236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5088.762453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5091.007149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5093.323481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5095.749608] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5096.387846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5104.490849] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 5109.441445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5110.625762] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5110.645756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5110.778982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5113.409558] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5119.374738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5119.453860] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5119.475319] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5119.627335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5122.523252] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5126.983764] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5130.092576] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5134.118534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5134.179700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5134.311292] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5134.423550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5137.049087] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5142.201385] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5146.082937] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5146.087341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5146.092982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5146.592776] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5146.603293] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5146.613053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5149.555462] Lustre: server umount lustre-OST0000 complete [ 5151.712540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5151.713711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5151.718287] Lustre: Skipped 1 previous similar message [ 5156.834112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5156.839113] Lustre: Skipped 2 previous similar messages [ 5157.742755] Lustre: server umount lustre-MDT0000 complete [ 5160.304311] LustreError: 349023:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759847778 with bad export cookie 12946559176774143736 [ 5160.312259] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5160.315438] LustreError: 349023:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5160.513876] Lustre: server umount lustre-MDT0001 complete [ 5166.424608] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 5168.262544] Key type lgssc unregistered [ 5168.428701] LNet: 350740:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5168.433429] LNet: Removed LNI 192.168.202.117@tcp [ 5168.928625] Key type .llcrypt unregistered [ 5168.933144] Key type ._llcrypt unregistered [ 5182.007416] alg: No test for adler32 (adler32-zlib) [ 5182.760227] Key type ._llcrypt registered [ 5182.761717] Key type .llcrypt registered [ 5182.852931] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 5183.572888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5183.764728] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 5183.902709] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 5183.907032] LNet: Accept secure, port 988 [ 5185.543207] Key type lgssc registered [ 5186.105154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5190.413938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5191.635116] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5191.669660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5193.863354] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5196.768120] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5198.606555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5198.805610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5201.148850] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5205.194574] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5208.060679] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5211.535367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5211.720218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5213.843373] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5217.578331] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5227.491426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5227.496017] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5227.503335] Lustre: Skipped 2 previous similar messages [ 5227.506365] Lustre: Skipped 1 previous similar message [ 5228.157664] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.17@tcp (stopping) [ 5228.160508] Lustre: Skipped 1 previous similar message [ 5232.157548] Lustre: server umount lustre-MDT0000 complete [ 5232.611494] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5234.301143] LustreError: 352182:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759847852 with bad export cookie 380687180028537491 [ 5234.305976] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5234.306577] LustreError: 352182:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5237.730173] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5237.730746] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5237.738281] LustreError: Skipped 3 previous similar messages [ 5237.749729] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5239.777901] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5239.781912] Lustre: Skipped 1 previous similar message [ 5240.609815] Lustre: server umount lustre-MDT0001 complete [ 5244.439173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5248.865246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5251.042260] Lustre: Evicted from MGS (at 192.168.202.117@tcp) after server handle changed from 0x54878f928256a93 to 0x54878f928258669 [ 5251.048258] Lustre: MGC192.168.202.117@tcp: Connection restored to 192.168.202.117@tcp (at 0@lo) [ 5251.177158] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5251.184990] LustreError: Skipped 3 previous similar messages [ 5251.216551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5251.246515] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:193 [ 5253.282150] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5257.858641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5258.044531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5259.974191] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5261.092754] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 5263.303775] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5270.310566] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 5271.534959] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 5274.313141] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5275.153962] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5277.153650] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5277.157108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5277.163858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5277.166623] Lustre: Skipped 1 previous similar message [ 5278.690074] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5283.124743] Lustre: server umount lustre-OST0000 complete [ 5285.857712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5285.860130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5285.870360] Lustre: Skipped 2 previous similar messages [ 5285.873101] Lustre: Skipped 3 previous similar messages [ 5291.280820] Lustre: server umount lustre-MDT0000 complete [ 5293.058683] LustreError: 352183:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759847911 with bad export cookie 380687180028544617 [ 5293.060574] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5293.065377] LustreError: 352183:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5299.477203] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 5300.850421] Key type lgssc unregistered [ 5300.981687] LNet: 358692:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5300.986361] LNet: Removed LNI 192.168.202.117@tcp [ 5301.363744] Key type .llcrypt unregistered [ 5301.365406] Key type ._llcrypt unregistered [ 5313.490095] alg: No test for adler32 (adler32-zlib) [ 5314.242474] Key type ._llcrypt registered [ 5314.243742] Key type .llcrypt registered [ 5314.296087] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 5314.798809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5315.003195] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 5315.108878] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 5315.112337] LNet: Accept secure, port 988 [ 5316.720418] Key type lgssc registered [ 5317.292087] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5321.302238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5322.534635] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5322.567420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5324.548001] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5327.841988] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5328.559800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5328.779940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5330.770736] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5334.275423] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5337.130239] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5339.106095] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5339.110464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5339.117374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5343.202701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5343.206911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5344.227953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5344.229841] Lustre: Skipped 1 previous similar message [ 5344.733373] Lustre: server umount lustre-MDT0000 complete [ 5346.406585] LustreError: 360132:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759847964 with bad export cookie 16386065494004837031 [ 5346.409329] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5346.410924] LustreError: 360132:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5346.558375] Lustre: server umount lustre-MDT0001 complete [ 5353.731683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5353.898418] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5353.919553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5354.045555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5356.080183] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5360.341475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5360.410696] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5360.432220] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5360.570982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5362.687144] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5366.932177] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5370.038330] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5373.686750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5373.788193] LustreError: 13b-9: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5373.793325] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5373.798419] LustreError: 364841:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 5373.801960] LustreError: 364841:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 5373.806232] LustreError: 364841:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 5373.880210] Lustre: server umount lustre-OST0000 complete [ 5373.881622] LustreError: 364841:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5376.879330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5376.970267] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5377.076713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5378.994436] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5382.437668] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5385.743642] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5385.776229] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5385.779693] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 5385.813311] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5385.842716] Lustre: lustre-OST0001: new disk, initializing [ 5385.845056] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5385.899486] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5385.925649] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:225 [ 5387.748749] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5391.173134] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5391.339711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5391.343905] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5396.448254] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5396.449774] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5396.453492] LustreError: Skipped 1 previous similar message [ 5396.461333] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5396.463732] Lustre: Skipped 1 previous similar message [ 5398.518201] Lustre: server umount lustre-OST0001 complete [ 5401.568468] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5401.570409] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5401.571566] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5401.574175] LustreError: Skipped 1 previous similar message [ 5401.579601] Lustre: Skipped 1 previous similar message [ 5401.580374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5401.596893] Lustre: Skipped 1 previous similar message [ 5406.444508] Lustre: server umount lustre-OST0000 complete [ 5411.808335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5411.808912] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5411.813128] Lustre: Skipped 1 previous similar message [ 5411.818169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5411.822110] Lustre: Skipped 1 previous similar message [ 5416.153394] LustreError: 362842:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759848034 with bad export cookie 16386065494004837570 [ 5416.156170] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5416.158598] LustreError: 362842:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5416.300202] Lustre: server umount lustre-MDT0001 complete [ 5416.302065] Lustre: Skipped 1 previous similar message [ 5421.374900] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 5422.649264] Key type lgssc unregistered [ 5422.785510] LNet: 368203:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5422.789207] LNet: Removed LNI 192.168.202.117@tcp [ 5423.146636] Key type .llcrypt unregistered [ 5423.149042] Key type ._llcrypt unregistered [ 5437.941366] alg: No test for adler32 (adler32-zlib) [ 5438.693473] Key type ._llcrypt registered [ 5438.695085] Key type .llcrypt registered [ 5438.741482] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 5446.580953] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 5447.129575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5447.283080] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 5447.394323] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 5447.397748] LNet: Accept secure, port 988 [ 5448.991123] Key type lgssc registered [ 5449.426512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5453.276104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5456.758137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5459.652254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5462.910560] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5466.421117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5466.463879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5467.603049] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5467.629308] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5467.673648] Lustre: lustre-MDT0000: new disk, initializing [ 5467.716604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5467.725712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5469.876240] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5475.746364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5475.784226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5475.829879] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5475.852319] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5475.855314] Lustre: Skipped 1 previous similar message [ 5475.913682] Lustre: lustre-MDT0001: new disk, initializing [ 5475.979906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5476.005664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5476.012317] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5477.971122] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5483.735801] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5486.704538] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5489.837405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5489.870545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5489.874523] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5490.008900] Lustre: lustre-OST0000: new disk, initializing [ 5490.011534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5490.061626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5492.225939] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5497.410783] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5498.379526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5498.385751] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5499.393331] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5499.500822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5501.612536] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5501.709696] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5503.458074] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5503.461284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5503.461542] LustreError: Skipped 1 previous similar message [ 5503.461906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5503.470912] Lustre: Skipped 1 previous similar message [ 5504.698794] Lustre: server umount lustre-OST0000 complete [ 5507.041397] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5507.049037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5507.059150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5507.062289] Lustre: Skipped 1 previous similar message [ 5508.577544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5508.577725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5512.591475] Lustre: server umount lustre-MDT0000 complete [ 5513.697141] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5513.703051] LustreError: Skipped 1 previous similar message [ 5514.162453] LustreError: 375970:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759848132 with bad export cookie 16564344045033908332 [ 5514.165297] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5514.167734] LustreError: 375970:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5514.291671] Lustre: server umount lustre-MDT0001 complete [ 5517.740809] Lustre: DEBUG MARKER: == conf-sanity test 60: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 10:42:15 (1759848135) [ 5520.280670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5522.720112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5532.655309] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 5536.208851] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 5540.291753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5543.087919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5545.739958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5548.193822] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5551.002561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5551.037242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5551.145332] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5551.160712] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5551.202125] Lustre: lustre-MDT0000: new disk, initializing [ 5551.237557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5551.246523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5552.966727] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5557.922330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5557.961241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5558.010151] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5558.033757] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5558.036488] Lustre: Skipped 1 previous similar message [ 5558.079216] Lustre: lustre-MDT0001: new disk, initializing [ 5558.119303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5558.136418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5558.143151] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5559.972753] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5564.872445] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5567.597817] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5570.551193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5570.581832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5570.586281] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5570.701720] Lustre: lustre-OST0000: new disk, initializing [ 5570.705084] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5570.746654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5572.580336] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5576.191710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5576.196528] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5577.231182] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5579.143744] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5579.260853] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5581.032433] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5581.124234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5586.400491] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5586.401418] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5586.404026] LustreError: Skipped 1 previous similar message [ 5586.409332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5586.414195] Lustre: Skipped 2 previous similar messages [ 5591.520821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5591.523315] Lustre: Skipped 1 previous similar message [ 5596.127128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5596.192076] Lustre: server umount lustre-OST0000 complete [ 5599.201327] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5599.204755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5599.209376] Lustre: Skipped 1 previous similar message [ 5601.760047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5601.762839] Lustre: Skipped 2 previous similar messages [ 5603.784539] Lustre: server umount lustre-MDT0000 complete [ 5605.387976] LustreError: 380044:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759848223 with bad export cookie 16564344045033909172 [ 5605.391144] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5605.392977] LustreError: 380044:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5609.058232] Lustre: DEBUG MARKER: == conf-sanity test 61: large xattr ====================== 10:43:46 (1759848226) [ 5612.925327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5613.130403] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5613.163747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5614.979191] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5618.363594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5620.201627] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5623.153779] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5625.489160] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5628.179481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5629.998105] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5633.069598] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5647.694785] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61.conf-sanity [ 5648.407941] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61.conf-sanity [ 5649.076405] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61.conf-sanity [ 5649.778166] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61.conf-sanity after remounting MDS [ 5650.574463] Lustre: Failing over lustre-MDT0000 [ 5650.719280] Lustre: server umount lustre-MDT0000 complete [ 5650.720842] Lustre: Skipped 1 previous similar message [ 5653.985979] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5653.987452] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5653.991297] Lustre: Skipped 3 previous similar messages [ 5653.995806] LustreError: Skipped 1 previous similar message [ 5659.104160] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5659.108954] LustreError: Skipped 4 previous similar messages [ 5661.151257] Lustre: 370520:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759848272/real 1759848272] req@000000001896adf3 x1845334444647936/t0(0) o400->MGC192.168.202.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1759848279 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5661.158361] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5664.581681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5666.272714] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5666.277317] LustreError: Skipped 7 previous similar messages [ 5667.297928] Lustre: Evicted from MGS (at 192.168.202.117@tcp) after server handle changed from 0xe5e05f25b59a6efc to 0xe5e05f25b59a72d0 [ 5667.304181] LustreError: 388039:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.202.117@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000985a0afe) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5667.311368] Lustre: MGC192.168.202.117@tcp: Connection restored to 192.168.202.117@tcp (at 0@lo) [ 5667.402875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5667.406055] Lustre: Skipped 2 previous similar messages [ 5667.440859] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5669.089949] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5672.416182] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5672.419935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 5672.563303] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5672.577544] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 5674.718117] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5675.355653] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5677.348894] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61.conf-sanity [ 5678.109535] Lustre: Failing over lustre-MDT0000 [ 5678.242381] Lustre: server umount lustre-MDT0000 complete [ 5682.656798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5682.658589] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5682.662526] Lustre: Skipped 2 previous similar messages [ 5682.669395] LustreError: Skipped 6 previous similar messages [ 5688.799166] Lustre: 370520:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759848300/real 1759848300] req@000000001c848eba x1845334444656192/t0(0) o400->MGC192.168.202.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1759848307 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5688.807727] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5692.045423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5694.945432] Lustre: Evicted from MGS (at 192.168.202.117@tcp) after server handle changed from 0xe5e05f25b59a72d0 to 0xe5e05f25b59a75d9 [ 5694.950437] Lustre: MGC192.168.202.117@tcp: Connection restored to (at 0@lo) [ 5694.952741] Lustre: Skipped 2 previous similar messages [ 5695.083284] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5695.088071] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5696.789379] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5700.068693] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 5700.076662] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5700.097296] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [ 5702.389796] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5703.036131] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5710.306402] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5710.307110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5710.314472] Lustre: Skipped 1 previous similar message [ 5711.827475] Lustre: server umount lustre-MDT0000 complete [ 5713.375727] LustreError: 384679:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759848331 with bad export cookie 16564344045033911769 [ 5713.377351] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5713.379747] LustreError: 384679:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5721.567129] Lustre: 390500:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759848333/real 1759848333] req@00000000e29f2883 x1845334444665856/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759848339 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5727.517556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5727.706664] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5727.714692] LustreError: Skipped 16 previous similar messages [ 5729.451813] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5732.918184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5733.064625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5733.067266] Lustre: Skipped 2 previous similar messages [ 5734.891347] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5738.023658] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5740.341753] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5743.037355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5744.165112] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [ 5744.775215] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5747.708725] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5754.336354] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5754.340057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5754.344426] Lustre: Skipped 3 previous similar messages [ 5754.346418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5776.351200] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5776.413791] Lustre: server umount lustre-MDT0000 complete [ 5776.415380] Lustre: Skipped 3 previous similar messages [ 5778.125617] LustreError: 391307:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759848396 with bad export cookie 16564344045033912511 [ 5778.129535] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5778.130306] LustreError: 391307:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5783.789978] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 5785.001659] Key type lgssc unregistered [ 5785.140492] LNet: 394977:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5785.143611] LNet: Removed LNI 192.168.202.117@tcp [ 5785.479818] Key type .llcrypt unregistered [ 5785.482468] Key type ._llcrypt unregistered [ 5796.445752] alg: No test for adler32 (adler32-zlib) [ 5797.198297] Key type ._llcrypt registered [ 5797.199808] Key type .llcrypt registered [ 5797.243735] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 5797.665757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5797.888945] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 5798.003177] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 5798.005657] LNet: Accept secure, port 988 [ 5799.599174] Key type lgssc registered [ 5800.008836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5803.098310] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5803.101828] LustreError: 396460:0:(osd_handler.c:8214:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5803.106139] LustreError: 396460:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5803.109115] LustreError: 396460:0:(obd_mount.c:200:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5803.112497] LustreError: 396460:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5803.116697] LustreError: 396460:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5805.965622] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5805.969274] LustreError: 396857:0:(osd_handler.c:8214:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5805.973389] LustreError: 396857:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5805.976280] LustreError: 396857:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5805.979480] LustreError: 396857:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5805.983655] LustreError: 396857:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5812.486746] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 5813.640868] Key type lgssc unregistered [ 5813.760425] LNet: 397814:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5813.763944] LNet: Removed LNI 192.168.202.117@tcp [ 5814.112441] Key type .llcrypt unregistered [ 5814.113936] Key type ._llcrypt unregistered [ 5820.957859] alg: No test for adler32 (adler32-zlib) [ 5821.709396] Key type ._llcrypt registered [ 5821.711037] Key type .llcrypt registered [ 5821.760239] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 5828.213139] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 5828.622816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5828.808659] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 5828.906803] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 5828.909216] LNet: Accept secure, port 988 [ 5830.503123] Key type lgssc registered [ 5830.908137] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5834.592460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5837.294115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5839.591468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5841.844134] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5844.362552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5844.387565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5845.479297] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5845.491804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5845.532311] Lustre: lustre-MDT0000: new disk, initializing [ 5845.564822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5845.575604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5847.294572] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5851.868506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5851.904187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5851.947499] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5851.968461] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5851.971467] Lustre: Skipped 1 previous similar message [ 5852.012904] Lustre: lustre-MDT0001: new disk, initializing [ 5852.054960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5852.073082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5852.077638] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5853.852987] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5858.060389] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5860.416615] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5862.871521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5862.904221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5862.906827] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5863.003359] Lustre: lustre-OST0000: new disk, initializing [ 5863.005654] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5863.034297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5864.549110] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5868.720224] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5870.348887] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5871.099654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5871.103818] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5871.467632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5873.231785] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5873.320257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5876.192799] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5876.193214] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5876.197276] LustreError: Skipped 1 previous similar message [ 5876.202366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5876.206175] Lustre: Skipped 1 previous similar message [ 5880.151156] Lustre: server umount lustre-OST0000 complete [ 5882.847983] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5882.851223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5882.855819] Lustre: Skipped 1 previous similar message [ 5882.858322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5886.431948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5886.432300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5887.685018] Lustre: server umount lustre-MDT0000 complete [ 5889.166737] LustreError: 404770:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759848507 with bad export cookie 470695745188286750 [ 5889.169460] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5889.171139] LustreError: 404770:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5889.300167] Lustre: server umount lustre-MDT0001 complete [ 5892.510180] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5893.168276] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 10:48:30 (1759848510) [ 5895.755200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5895.964360] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5895.996124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5897.745714] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5901.154578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5901.280263] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5901.283440] Lustre: Skipped 1 previous similar message [ 5901.341671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5903.151391] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5905.918423] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5908.077227] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5910.759840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5910.917913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5912.592115] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5915.662423] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5918.607114] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5918.645611] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5918.650657] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 5918.714243] Lustre: lustre-OST0001: new disk, initializing [ 5918.716521] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5920.496607] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 5924.370382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5924.375601] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5924.816058] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5929.443399] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5929.443479] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5929.449353] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5929.454574] Lustre: Skipped 1 previous similar message [ 5932.910304] Lustre: server umount lustre-OST0001 complete [ 5934.562538] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5934.567674] LustreError: Skipped 1 previous similar message [ 5939.682602] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5939.687054] LustreError: Skipped 1 previous similar message [ 5944.810102] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5944.814535] LustreError: Skipped 1 previous similar message [ 5946.848373] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5946.851608] LustreError: Skipped 1 previous similar message [ 5946.853356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5946.858439] Lustre: Skipped 1 previous similar message [ 5946.860535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5946.863311] Lustre: Skipped 1 previous similar message [ 5949.921596] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5949.926680] LustreError: Skipped 1 previous similar message [ 5951.462607] Lustre: server umount lustre-OST0000 complete [ 5955.040556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5955.045973] Lustre: Skipped 2 previous similar messages [ 5960.164046] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5960.171074] LustreError: Skipped 1 previous similar message [ 5960.878355] LustreError: 408595:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759848579 with bad export cookie 470695745188287590 [ 5960.881170] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5960.882777] LustreError: 408595:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5967.113307] Lustre: server umount lustre-MDT0001 complete [ 5967.115302] Lustre: Skipped 1 previous similar message [ 5972.327766] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 5973.545297] Key type lgssc unregistered [ 5973.688698] LNet: 410588:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5973.693283] LNet: Removed LNI 192.168.202.117@tcp [ 5974.061899] Key type .llcrypt unregistered [ 5974.063670] Key type ._llcrypt unregistered [ 5988.004185] alg: No test for adler32 (adler32-zlib) [ 5988.755369] Key type ._llcrypt registered [ 5988.756785] Key type .llcrypt registered [ 5988.816768] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 5995.603145] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 5995.966865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5996.226892] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 5996.325250] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 5996.327850] LNet: Accept secure, port 988 [ 5997.911081] Key type lgssc registered [ 5998.345117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6002.002410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6004.857252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6007.240619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6009.802806] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6012.690590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6012.721895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6013.819547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6013.832144] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6013.873319] Lustre: lustre-MDT0000: new disk, initializing [ 6013.899584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6013.908524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6015.691749] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6020.661651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6020.698440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6020.742925] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6020.766764] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6020.769895] Lustre: Skipped 1 previous similar message [ 6020.810465] Lustre: lustre-MDT0001: new disk, initializing [ 6020.844504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6020.859337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6020.863728] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6022.604112] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6026.805662] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6028.917056] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6031.272441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6031.299728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6031.303802] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6031.405123] Lustre: lustre-OST0000: new disk, initializing [ 6031.407559] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6031.434119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6032.965865] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6035.390179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6035.393899] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6037.236804] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6038.884303] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6038.966511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6040.658380] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6040.747330] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6045.664822] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6045.665850] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6045.668634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6045.672126] Lustre: Skipped 1 previous similar message [ 6047.662282] Lustre: server umount lustre-OST0000 complete [ 6050.784570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6050.785308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6050.789105] Lustre: Skipped 1 previous similar message [ 6050.793991] Lustre: Skipped 2 previous similar messages [ 6055.365536] Lustre: server umount lustre-MDT0000 complete [ 6055.905823] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6055.911903] LustreError: Skipped 1 previous similar message [ 6057.053194] LustreError: 414687:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759848675 with bad export cookie 1229956027947974097 [ 6057.056302] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6057.058044] LustreError: 414687:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6057.196545] Lustre: server umount lustre-MDT0001 complete [ 6060.819021] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 10:51:18 (1759848678) [ 6064.607167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6067.892756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6068.077437] Lustre: lustre-MDT0000: new disk, initializing [ 6068.091135] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6068.116617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6069.708311] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6073.134813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6073.290603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6075.050137] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6078.133055] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6080.310858] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6083.552862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6083.554378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6083.558247] Lustre: Skipped 1 previous similar message [ 6083.558587] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6083.558592] LustreError: Skipped 1 previous similar message [ 6087.237518] Lustre: server umount lustre-MDT0000 complete [ 6088.673551] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6088.684871] LustreError: Skipped 1 previous similar message [ 6088.733943] LustreError: 419814:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759848707 with bad export cookie 1229956027947974951 [ 6088.735407] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6088.739459] LustreError: 419814:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6092.686441] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 10:51:50 (1759848710) [ 6095.339109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6095.520225] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6095.551720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6097.160149] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6100.468827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6102.186153] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6105.015649] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6108.451646] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6110.907018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6112.561636] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6115.356543] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6128.943542] Lustre: Permanently deactivating lustre-OST0000 [ 6128.950688] Lustre: Setting parameter lustre-OST0000-osc.osc.active in log lustre-client [ 6138.865699] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6140.585467] LustreError: 425689:0:(mgs_llog.c:1551:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002020 [ 6140.589252] LustreError: 425689:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 6140.591873] LustreError: 425689:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 6141.387254] LustreError: 425737:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 6141.390323] LustreError: 425737:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 6148.324489] Lustre: server umount lustre-OST0000 complete [ 6148.326847] Lustre: Skipped 1 previous similar message [ 6149.506104] LustreError: 425988:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 6149.508942] LustreError: 425988:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 6152.160810] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6152.161162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6152.164783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6152.169134] Lustre: Skipped 1 previous similar message [ 6152.171841] Lustre: Skipped 1 previous similar message [ 6156.255165] Lustre: 426085:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759848768/real 1759848768] req@00000000defc61e0 x1845335020311808/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1759848774 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6156.328846] Lustre: server umount lustre-MDT0000 complete [ 6157.280352] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6157.286107] LustreError: Skipped 1 previous similar message [ 6157.705886] LustreError: 422410:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759848775 with bad export cookie 1229956027947975490 [ 6157.708192] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6157.712826] LustreError: 422410:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6163.935164] Lustre: 426287:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759848776/real 1759848776] req@00000000595bdd30 x1845335020312832/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1759848782 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6167.011042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6168.518541] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6169.471920] LustreError: 427196:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -6 [ 6171.633828] Lustre: 427541:0:(mgs_llog.c:1311:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.117@tcp [ 6177.759173] Lustre: 427640:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759848790/real 1759848790] req@000000001e3de773 x1845335020313216/t0(0) o251->MGC192.168.202.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1759848796 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6177.772990] Lustre: server umount MGS complete [ 6177.774513] Lustre: Skipped 1 previous similar message [ 6181.925518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6182.111924] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6182.118943] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6182.121422] Lustre: Skipped 1 previous similar message [ 6182.144877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6182.146931] Lustre: Skipped 2 previous similar messages [ 6183.867729] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6186.929248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6187.025550] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6188.563911] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6191.199114] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6193.134294] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6193.798450] Lustre: Permanently reactivating lustre-OST0000 [ 6193.803602] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 6193.805666] Lustre: Skipped 2 previous similar messages [ 6209.704453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6211.507084] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6214.286549] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6227.424298] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6227.425997] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6227.427204] LustreError: Skipped 1 previous similar message [ 6227.432068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6227.435639] Lustre: Skipped 1 previous similar message [ 6228.305507] Lustre: server umount lustre-OST0000 complete [ 6232.545051] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6232.548870] LustreError: Skipped 1 previous similar message [ 6232.911884] LustreError: 429052:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759848851 with bad export cookie 1229956027947976435 [ 6232.914153] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6232.914891] LustreError: 429052:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6237.671619] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 6238.675902] Key type lgssc unregistered [ 6238.791352] LNet: 432578:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6238.794454] LNet: Removed LNI 192.168.202.117@tcp [ 6239.091151] Key type .llcrypt unregistered [ 6239.091982] Key type ._llcrypt unregistered [ 6245.509044] alg: No test for adler32 (adler32-zlib) [ 6246.261378] Key type ._llcrypt registered [ 6246.262830] Key type .llcrypt registered [ 6246.300351] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 6252.366245] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 6252.778882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6252.945762] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 6253.017792] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 6253.019358] LNet: Accept secure, port 988 [ 6254.599131] Key type lgssc registered [ 6254.957383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6258.002257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6260.315265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6262.484079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6264.654757] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6267.107286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6267.130338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6268.225858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6268.237485] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6268.278268] Lustre: lustre-MDT0000: new disk, initializing [ 6268.311819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6268.319343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6269.840314] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6273.997138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6274.019608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6274.048674] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6274.065598] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6274.067471] Lustre: Skipped 1 previous similar message [ 6274.100872] Lustre: lustre-MDT0001: new disk, initializing [ 6274.130513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6274.141365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6274.144413] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6275.561159] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6279.287111] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6281.108694] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6283.255791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6283.283783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6283.287419] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6283.374417] Lustre: lustre-OST0000: new disk, initializing [ 6283.376114] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6283.401307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6284.726506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6284.730069] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6284.844333] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6288.664668] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6290.144974] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6290.224431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6291.605353] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6291.681627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6295.008627] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6295.008662] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6295.012242] LustreError: Skipped 1 previous similar message [ 6295.016739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6295.019629] Lustre: Skipped 1 previous similar message [ 6298.450130] Lustre: server umount lustre-OST0000 complete [ 6299.615582] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6299.617642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6299.620125] Lustre: Skipped 1 previous similar message [ 6299.621711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6305.247822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6305.249559] Lustre: Skipped 3 previous similar messages [ 6305.725640] Lustre: server umount lustre-MDT0000 complete [ 6306.980207] LustreError: 435840:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759848925 with bad export cookie 15992850469002369941 [ 6306.983327] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6306.984220] LustreError: 435840:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6307.099696] Lustre: server umount lustre-MDT0001 complete [ 6309.917411] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6310.459698] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 10:55:28 (1759848928) [ 6312.706795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6312.877559] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6312.899906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6314.221693] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6317.090651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6317.203680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6318.725970] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6321.549949] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6323.574229] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6325.958160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6326.084779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6327.622903] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6330.135378] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6330.856634] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6332.376961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6337.505479] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6337.505525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6337.506244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6337.516747] Lustre: Skipped 2 previous similar messages [ 6339.797070] Lustre: server umount lustre-OST0000 complete [ 6342.624383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6342.624932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6342.628095] Lustre: Skipped 1 previous similar message [ 6342.630439] Lustre: Skipped 1 previous similar message [ 6347.203292] Lustre: server umount lustre-MDT0000 complete [ 6347.743792] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6347.748562] LustreError: Skipped 1 previous similar message [ 6348.425900] LustreError: 443196:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759848966 with bad export cookie 15992850469002370781 [ 6348.428534] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6348.431055] LustreError: 443196:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6353.207422] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 6354.244522] Key type lgssc unregistered [ 6354.355414] LNet: 444290:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6354.359384] LNet: Removed LNI 192.168.202.117@tcp [ 6354.657185] Key type .llcrypt unregistered [ 6354.658772] Key type ._llcrypt unregistered [ 6364.051260] alg: No test for adler32 (adler32-zlib) [ 6364.802480] Key type ._llcrypt registered [ 6364.804044] Key type .llcrypt registered [ 6364.840186] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 6364.938384] Key type .llcrypt unregistered [ 6364.940235] Key type ._llcrypt unregistered [ 6371.937421] alg: No test for adler32 (adler32-zlib) [ 6372.689324] Key type ._llcrypt registered [ 6372.690284] Key type .llcrypt registered [ 6372.728400] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 6373.152057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6373.299439] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 6373.377677] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 6373.379280] LNet: Accept secure, port 988 [ 6374.959107] Key type lgssc registered [ 6375.315882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6378.097217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6379.241504] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6379.256380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6380.775693] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6383.795883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6383.909900] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6383.915367] LustreError: Skipped 1 previous similar message [ 6383.917540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6384.928520] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6384.933774] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 6384.934307] LustreError: Skipped 1 previous similar message [ 6385.456722] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6388.234993] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6390.241922] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6390.250249] LustreError: Skipped 1 previous similar message [ 6390.564480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6390.684636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6392.224092] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6400.479742] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6400.482677] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6400.486543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6400.992312] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6400.994326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6400.998270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6405.401534] Lustre: server umount lustre-OST0000 complete [ 6410.209600] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6410.212983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6410.218344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6412.874928] Lustre: server umount lustre-MDT0000 complete [ 6414.264232] LustreError: 446597:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759849032 with bad export cookie 10565812711442958095 [ 6414.267737] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6414.268512] LustreError: 446597:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6414.399153] Lustre: server umount lustre-MDT0001 complete [ 6419.495933] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 6420.463216] Key type lgssc unregistered [ 6420.577375] LNet: 449882:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6420.580949] LNet: Removed LNI 192.168.202.117@tcp [ 6420.885834] Key type .llcrypt unregistered [ 6420.887038] Key type ._llcrypt unregistered [ 6430.159198] alg: No test for adler32 (adler32-zlib) [ 6430.910441] Key type ._llcrypt registered [ 6430.911868] Key type .llcrypt registered [ 6430.956637] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 6431.450854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6431.606918] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 6431.693096] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 6431.695237] LNet: Accept secure, port 988 [ 6433.287121] Key type lgssc registered [ 6433.660623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6436.423046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6453.919325] LustreError: 451315:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.117@tcp: failed processing log, type 1: rc = -5 [ 6484.639192] LustreError: 451315:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.117@tcp: failed processing log, type 4: rc = -110 [ 6514.335181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6515.881733] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6518.689137] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6521.283059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6536.992198] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6537.027084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6537.055469] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [ 6538.744272] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6542.093260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6542.251034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6543.999891] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6546.843762] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6548.951803] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6552.544969] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6552.545442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6552.546938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6552.547880] LustreError: Skipped 1 previous similar message [ 6552.555189] Lustre: Skipped 1 previous similar message [ 6557.410739] Lustre: server umount lustre-OST0000 complete [ 6561.761512] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6561.764496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6561.769561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6561.771823] Lustre: Skipped 1 previous similar message [ 6562.784052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6562.784399] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6564.994510] Lustre: server umount lustre-MDT0000 complete [ 6566.321914] LustreError: 451323:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759849184 with bad export cookie 14384447832078016983 [ 6566.325106] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6566.325539] LustreError: 451323:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6566.448596] Lustre: server umount lustre-MDT0001 complete [ 6571.342639] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 6572.383571] Key type lgssc unregistered [ 6572.501299] LNet: 455009:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6572.503944] LNet: Removed LNI 192.168.202.117@tcp [ 6572.790308] Key type .llcrypt unregistered [ 6572.792234] Key type ._llcrypt unregistered [ 6581.882524] alg: No test for adler32 (adler32-zlib) [ 6582.634383] Key type ._llcrypt registered [ 6582.635806] Key type .llcrypt registered [ 6582.674467] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 6582.994975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6583.231444] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 6583.320171] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 6583.322485] LNet: Accept secure, port 988 [ 6584.903147] Key type lgssc registered [ 6585.251023] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6587.953212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6589.090338] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6589.106258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6590.707041] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6594.313995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6594.452288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6596.312991] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6599.513576] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6601.857150] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6604.494306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6604.609248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6606.106108] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6608.936163] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6618.980518] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [ 6624.224750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6624.226026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6624.228668] Lustre: Skipped 2 previous similar messages [ 6624.230425] Lustre: Skipped 1 previous similar message [ 6624.883316] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.17@tcp (stopping) [ 6624.886021] Lustre: Skipped 1 previous similar message [ 6626.888616] Lustre: server umount lustre-MDT0000 complete [ 6627.713955] LustreError: 459137:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.17@tcp arrived at 1759849245 with bad export cookie 2741784610738634746 [ 6627.719613] LustreError: 459137:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6628.401925] LustreError: 459137:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759849246 with bad export cookie 2741784610738634095 [ 6628.405841] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6628.405963] LustreError: 459137:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6629.344194] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6629.344376] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6629.344573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6629.345261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6629.358725] LustreError: Skipped 1 previous similar message [ 6633.377244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6633.378033] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6633.385851] LustreError: Skipped 1 previous similar message [ 6634.531316] Lustre: server umount lustre-OST0000 complete [ 6637.009923] Lustre: server umount lustre-MDT0001 complete [ 6641.763860] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 6642.794605] Key type lgssc unregistered [ 6642.922377] LNet: 460293:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6642.925183] LNet: Removed LNI 192.168.202.117@tcp [ 6643.247835] Key type .llcrypt unregistered [ 6643.249065] Key type ._llcrypt unregistered [ 6652.879128] alg: No test for adler32 (adler32-zlib) [ 6653.630366] Key type ._llcrypt registered [ 6653.631740] Key type .llcrypt registered [ 6653.672701] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 6654.041605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6654.280572] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 6654.377508] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 6654.380296] LNet: Accept secure, port 988 [ 6655.967117] Key type lgssc registered [ 6656.348821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6659.216060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6660.371239] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6660.388278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6661.858415] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6664.738397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6664.851546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6666.291684] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6668.827997] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6670.807124] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6673.027786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6673.134284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6674.495727] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6676.195427] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:161 [ 6676.994496] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6681.569342] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6681.569439] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6681.569747] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6681.573850] Lustre: Skipped 1 previous similar message [ 6683.243705] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.17@tcp (stopping) [ 6683.246444] Lustre: Skipped 1 previous similar message [ 6686.688745] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6686.690463] Lustre: Skipped 1 previous similar message [ 6691.808137] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6691.810025] Lustre: Skipped 2 previous similar messages [ 6693.343137] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6693.377115] Lustre: server umount lustre-MDT0001 complete [ 6693.484241] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.202.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6696.928563] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6696.929235] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6696.929383] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6696.930046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6701.145124] Lustre: server umount lustre-OST0000 complete [ 6702.048126] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6708.703152] Lustre: 464813:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759849320/real 1759849320] req@000000007fcaf159 x1845335710248512/t0(0) o251->MGC192.168.202.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1759849326 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6708.772827] Lustre: server umount lustre-MDT0000 complete [ 6714.544348] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 6715.561965] Key type lgssc unregistered [ 6715.674392] LNet: 465567:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6715.677286] LNet: Removed LNI 192.168.202.117@tcp [ 6715.948228] Key type .llcrypt unregistered [ 6715.949566] Key type ._llcrypt unregistered [ 6725.008310] alg: No test for adler32 (adler32-zlib) [ 6725.759288] Key type ._llcrypt registered [ 6725.760630] Key type .llcrypt registered [ 6725.806786] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 6731.033131] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 6731.527741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6731.670823] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 6731.761240] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 6731.763653] LNet: Accept secure, port 988 [ 6733.351172] Key type lgssc registered [ 6733.750580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6736.818129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6739.060024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6741.127809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6743.202227] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6746.137843] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 6749.092137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6751.343534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6751.365639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6752.451571] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6752.463131] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6752.499609] Lustre: lustre-MDT0000: new disk, initializing [ 6752.524144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6752.530935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6754.079125] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6757.696805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6759.928105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6759.951375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6759.957170] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6760.047952] Lustre: lustre-OST0000: new disk, initializing [ 6760.050329] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6760.053240] Lustre: Skipped 1 previous similar message [ 6760.077078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6761.495477] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6765.368934] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6768.715791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6770.895374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6770.915404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6770.947053] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6770.964459] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6771.001444] Lustre: lustre-MDT0001: new disk, initializing [ 6771.023113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6771.026919] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6771.047766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6772.483276] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6776.163721] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 6776.234686] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6777.654848] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 6780.789549] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 3 sec [ 6783.090275] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.17@tcp (stopping) [ 6785.504921] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6785.505479] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6785.507783] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6785.513623] Lustre: Skipped 1 previous similar message [ 6788.203917] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.17@tcp (stopping) [ 6788.206078] Lustre: Skipped 1 previous similar message [ 6788.412523] Lustre: server umount lustre-MDT0001 complete [ 6790.624741] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6790.628714] LustreError: Skipped 1 previous similar message [ 6795.745105] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6795.750139] LustreError: Skipped 1 previous similar message [ 6800.864215] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6800.864396] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6800.874101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6805.984161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6805.984718] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6811.104513] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6815.199114] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6815.257288] Lustre: server umount lustre-OST0000 complete [ 6829.023655] Lustre: 473994:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759849441/real 1759849441] req@000000001d4f0df3 x1845335792041664/t0(0) o251->MGC192.168.202.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1759849447 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6829.174849] Lustre: server umount lustre-MDT0000 complete [ 6839.895544] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 6841.659615] Key type lgssc unregistered [ 6841.820655] LNet: 474751:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6841.827843] LNet: Removed LNI 192.168.202.117@tcp [ 6842.332296] Key type .llcrypt unregistered [ 6842.333595] Key type ._llcrypt unregistered [ 6874.887355] alg: No test for adler32 (adler32-zlib) [ 6875.638369] Key type ._llcrypt registered [ 6875.639737] Key type .llcrypt registered [ 6875.700723] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 6875.859207] Key type .llcrypt unregistered [ 6875.860970] Key type ._llcrypt unregistered [ 6884.476402] alg: No test for adler32 (adler32-zlib) [ 6885.228414] Key type ._llcrypt registered [ 6885.229876] Key type .llcrypt registered [ 6885.276001] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 6885.669300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6885.928768] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 6886.023593] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 6886.026692] LNet: Accept secure, port 988 [ 6887.623378] Key type lgssc registered [ 6888.100842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6891.479379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6894.287166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6896.619782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6899.409726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6899.449684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6900.568523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6900.585701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6900.634741] Lustre: lustre-MDT0000: new disk, initializing [ 6900.681484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6900.694247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6902.669540] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6907.796609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6907.826211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6907.874062] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6907.898157] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6907.901175] Lustre: Skipped 1 previous similar message [ 6907.947458] Lustre: lustre-MDT0001: new disk, initializing [ 6907.994728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6908.015034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6908.020710] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6910.183888] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6914.990127] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6917.700788] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6920.896728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6920.932681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6920.938145] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6921.066788] Lustre: lustre-OST0000: new disk, initializing [ 6921.069688] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6921.115409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6922.064596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6922.070692] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6923.204111] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6928.215030] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6932.449128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6932.451529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6932.455346] Lustre: Skipped 2 previous similar messages [ 6932.461172] Lustre: Skipped 2 previous similar messages [ 6937.523449] Lustre: server umount lustre-MDT0000 complete [ 6937.569693] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6937.574770] LustreError: Skipped 2 previous similar messages [ 6939.364964] LustreError: 478235:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759849557 with bad export cookie 11473590985174365995 [ 6939.365875] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6939.372716] LustreError: 478235:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6939.584641] Lustre: server umount lustre-MDT0001 complete [ 6946.783631] Lustre: 481490:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759849559/real 1759849559] req@000000003e3f179f x1845335953520960/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759849565 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6946.880642] Lustre: server umount lustre-OST0000 complete [ 6951.515286] Lustre: DEBUG MARKER: == conf-sanity test 73: failnode to update from mountdata properly ========================================================== 11:06:09 (1759849569) [ 6951.956446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6954.633222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6954.856098] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6954.884493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6956.552943] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6960.072839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6960.096230] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6960.244507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6961.979363] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6964.705591] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6966.785595] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6969.595284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6969.633840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6969.729275] Lustre: Found index 0 for lustre-OST0000, updating log [ 6969.803138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6970.853433] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5 to 0x0:33 [ 6971.602905] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 6975.045448] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6981.088482] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6981.089994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6981.091342] LustreError: Skipped 1 previous similar message [ 6981.092068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6981.095761] Lustre: Skipped 1 previous similar message [ 6981.100847] Lustre: Skipped 1 previous similar message [ 6982.949295] Lustre: server umount lustre-OST0000 complete [ 6985.695894] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6985.701170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6985.706771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6988.451035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6988.453678] Lustre: Skipped 3 previous similar messages [ 6990.543662] Lustre: server umount lustre-MDT0000 complete [ 6992.289431] LustreError: 482296:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759849610 with bad export cookie 11473590985174367332 [ 6992.290892] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6992.294429] LustreError: 482296:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6996.431657] Lustre: DEBUG MARKER: == conf-sanity test 75: The order of --index should be irrelevant ========================================================== 11:06:53 (1759849613) [ 6999.241609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7001.644313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7004.155394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7006.316612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7013.077921] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 7016.714943] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 7020.611564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7023.324706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7025.832731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7028.384136] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7031.008061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7031.035604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7031.140185] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7031.159256] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7031.199909] Lustre: lustre-MDT0000: new disk, initializing [ 7031.234712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7031.244270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7032.968670] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 7038.244653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7038.277153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7038.324059] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7038.353741] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7038.356361] Lustre: Skipped 1 previous similar message [ 7038.409414] Lustre: lustre-MDT0001: new disk, initializing [ 7038.464820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7038.469891] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7040.668548] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 7045.990534] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7049.356093] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7053.520900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7053.562527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7053.568702] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7053.698571] Lustre: lustre-OST0000: new disk, initializing [ 7053.701488] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7056.121711] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all 8 [ 7061.013631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7061.027726] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7061.291594] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7063.546424] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7063.655371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7065.705465] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7065.827688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7071.202176] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7071.202861] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7071.211209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7071.223265] Lustre: Skipped 2 previous similar messages [ 7071.228296] Lustre: Skipped 1 previous similar message [ 7072.895706] Lustre: server umount lustre-OST0000 complete [ 7072.898157] Lustre: Skipped 1 previous similar message [ 7076.320893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7076.328137] Lustre: Skipped 1 previous similar message [ 7081.441641] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7081.448379] LustreError: Skipped 1 previous similar message [ 7083.110276] LustreError: 489323:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759849701 with bad export cookie 11473590985174368382 [ 7083.113195] LustreError: 166-1: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7083.125780] LustreError: 489323:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7096.139799] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 7100.542370] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 7106.522890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7110.901661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7114.475476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7118.575499] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7119.586396] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6998 sec ========== 11:08:56 (1759849736) [ 7142.539439] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 7144.174556] Key type lgssc unregistered [ 7144.346757] LNet: 498055:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7144.352987] LNet: Removed LNI 192.168.202.117@tcp [ 7144.789456] Key type .llcrypt unregistered [ 7144.791786] Key type ._llcrypt unregistered