[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 487413484 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.032 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003208] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229855db145, max_idle_ns: 440795257019 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4800.06 BogoMIPS (lpj=2400032) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010125] LSM: Security Framework initializing [ 0.011043] Yama: becoming mindful. [ 0.012032] SELinux: Initializing. [ 0.014062] *** VALIDATE selinux *** [ 0.022140] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026522] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028142] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030118] *** VALIDATE tmpfs *** [ 0.031138] *** VALIDATE proc *** [ 0.032235] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.034250] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038025] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.042497] debug: unmapping init [mem 0xffffffffb7059000-0xffffffffb7060fff] [ 0.044291] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045679] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046020] ... version: 2 [ 0.047010] ... bit width: 48 [ 0.048008] ... generic registers: 4 [ 0.049009] ... value mask: 0000ffffffffffff [ 0.050010] ... max period: 00007fffffffffff [ 0.051013] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.053377] rcu: Hierarchical SRCU implementation. [ 0.055414] smp: Bringing up secondary CPUs ... [ 0.056370] x86: Booting SMP configuration: [ 0.057017] .... node #0, CPUs: #1 #2 #3 [ 0.060014] smp: Brought up 1 node, 4 CPUs [ 0.062010] smpboot: Max logical packages: 1 [ 0.063013] smpboot: Total of 4 processors activated (19200.25 BogoMIPS) [ 0.132531] node 0 deferred pages initialised in 68ms [ 0.136428] devtmpfs: initialized [ 0.138272] x86/mm: Memory block size: 128MB [ 0.141717] gcov: version magic: 0x41383552 [ 0.144285] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.148092] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151275] pinctrl core: initialized pinctrl subsystem [ 0.153288] [ 0.153856] ************************************************************* [ 0.157012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159011] ** ** [ 0.162012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165013] ** ** [ 0.167012] ** This means that this kernel is built to expose internal ** [ 0.170021] ** IOMMU data structures, which may compromise security on ** [ 0.173017] ** your system. ** [ 0.176010] ** ** [ 0.179020] ** If you see this message and you are not debugging the ** [ 0.182013] ** kernel, report this immediately to your vendor! ** [ 0.184011] ** ** [ 0.187013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.189011] ************************************************************* [ 0.192624] NET: Registered protocol family 16 [ 0.194424] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.197061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.200072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.203129] cpuidle: using governor menu [ 0.204740] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.207649] PCI: Using configuration type 1 for base access [ 0.209271] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.219210] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.221022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.223109] cryptd: max_cpu_qlen set to 1000 [ 0.224303] ACPI: Added _OSI(Module Device) [ 0.227020] ACPI: Added _OSI(Processor Device) [ 0.229019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.231021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.237723] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.246318] ACPI: Interpreter enabled [ 0.248291] ACPI: PM: (supports S0 S3 S4 S5) [ 0.250020] ACPI: Using IOAPIC for interrupt routing [ 0.252136] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.256480] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.267764] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.270062] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.273032] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.277105] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.282519] acpiphp: Slot [2] registered [ 0.284312] acpiphp: Slot [5] registered [ 0.287025] acpiphp: Slot [6] registered [ 0.289220] acpiphp: Slot [7] registered [ 0.291125] acpiphp: Slot [8] registered [ 0.293120] acpiphp: Slot [9] registered [ 0.295179] acpiphp: Slot [10] registered [ 0.296118] acpiphp: Slot [3] registered [ 0.298095] acpiphp: Slot [4] registered [ 0.299111] acpiphp: Slot [11] registered [ 0.301090] acpiphp: Slot [12] registered [ 0.302083] acpiphp: Slot [13] registered [ 0.304094] acpiphp: Slot [14] registered [ 0.305175] acpiphp: Slot [15] registered [ 0.307088] acpiphp: Slot [16] registered [ 0.308152] acpiphp: Slot [17] registered [ 0.310089] acpiphp: Slot [18] registered [ 0.312091] acpiphp: Slot [19] registered [ 0.313077] acpiphp: Slot [20] registered [ 0.315181] acpiphp: Slot [21] registered [ 0.316115] acpiphp: Slot [22] registered [ 0.317110] acpiphp: Slot [23] registered [ 0.319223] acpiphp: Slot [24] registered [ 0.321185] acpiphp: Slot [25] registered [ 0.323162] acpiphp: Slot [26] registered [ 0.325145] acpiphp: Slot [27] registered [ 0.327139] acpiphp: Slot [28] registered [ 0.329136] acpiphp: Slot [29] registered [ 0.330134] acpiphp: Slot [30] registered [ 0.333174] acpiphp: Slot [31] registered [ 0.335085] PCI host bridge to bus 0000:00 [ 0.338037] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.341038] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.344041] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.347036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.351023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.353255] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.356210] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.358911] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.362370] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.371865] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.377052] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.382029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.385016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.388017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.391389] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.394826] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.398061] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.400915] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.406024] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.420973] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.426019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.432950] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.446023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.454022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.473024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.486599] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.495023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.502022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.524026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.533000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.541020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.550023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.571024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.581154] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.587019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.595025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.612018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.624458] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.635024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.643015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.661014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.671623] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.678017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.684013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.701015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.714170] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.716378] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.718527] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.722707] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.726302] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.734058] iommu: Default domain type: Passthrough [ 0.736564] SCSI subsystem initialized [ 0.738137] ACPI: bus type USB registered [ 0.739096] usbcore: registered new interface driver usbfs [ 0.741056] usbcore: registered new interface driver hub [ 0.743078] usbcore: registered new device driver usb [ 0.744142] pps_core: LinuxPPS API ver. 1 registered [ 0.745010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.747048] PTP clock support registered [ 0.748146] EDAC MC: Ver: 3.0.0 [ 0.750224] PCI: Using ACPI for IRQ routing [ 0.753211] NetLabel: Initializing [ 0.754016] NetLabel: domain hash size = 128 [ 0.756017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.758106] NetLabel: unlabeled traffic allowed by default [ 0.761076] vgaarb: loaded [ 0.763299] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.764010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.769653] clocksource: Switched to clocksource kvm-clock [ 0.882646] VFS: Disk quotas dquot_6.6.0 [ 0.884365] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.887777] *** VALIDATE ramfs *** [ 0.889289] *** VALIDATE hugetlbfs *** [ 0.891332] pnp: PnP ACPI init [ 0.893799] pnp: PnP ACPI: found 6 devices [ 0.914845] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.917947] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.919886] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.921955] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.924204] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.926340] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.929171] NET: Registered protocol family 2 [ 0.931618] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.936476] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.939785] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.944819] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.948169] TCP: Hash tables configured (established 65536 bind 65536) [ 0.951100] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.954271] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.957071] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.960133] NET: Registered protocol family 1 [ 0.962631] RPC: Registered named UNIX socket transport module. [ 0.965478] RPC: Registered udp transport module. [ 0.967542] RPC: Registered tcp transport module. [ 0.969509] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.971878] NET: Registered protocol family 44 [ 0.973384] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.975370] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.977265] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.979372] PCI: CLS 0 bytes, default 64 [ 0.980935] Unpacking initramfs... [ 2.378785] debug: unmapping init [mem 0xffff8a98fcc54000-0xffff8a98fffbffff] [ 2.383365] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.387332] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.390547] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229855db145, max_idle_ns: 440795257019 ns [ 2.874205] Initialise system trusted keyrings [ 2.875807] Key type blacklist registered [ 2.878080] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.887798] zbud: loaded [ 2.891303] *** VALIDATE nfs *** [ 2.892910] *** VALIDATE nfs4 *** [ 2.894897] pstore: using deflate compression [ 2.899210] Platform Keyring initialized [ 2.999620] NET: Registered protocol family 38 [ 3.001156] Key type asymmetric registered [ 3.002643] Asymmetric key parser 'x509' registered [ 3.004474] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.007259] io scheduler mq-deadline registered [ 3.008753] io scheduler kyber registered [ 3.010251] io scheduler bfq registered [ 3.013165] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.016019] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.018585] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.022135] ACPI: Power Button [PWRF] [ 3.028511] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.039771] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.052271] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.060270] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.087753] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.116932] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.148642] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.158199] Non-volatile memory driver v1.3 [ 3.160218] Linux agpgart interface v0.103 [ 3.193873] virtio_blk virtio1: [vda] 68040 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.197098] vda: detected capacity change from 0 to 34836480 [ 3.215536] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.219695] vdb: detected capacity change from 0 to 1073741824 [ 3.235665] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.241036] vdc: detected capacity change from 0 to 2621440000 [ 3.258133] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.261208] vdd: detected capacity change from 0 to 2621440000 [ 3.275909] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.278818] vde: detected capacity change from 0 to 4294967296 [ 3.294205] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.297858] vdf: detected capacity change from 0 to 4294967296 [ 3.305621] libphy: Fixed MDIO Bus: probed [ 3.317199] usbcore: registered new interface driver usbserial_generic [ 3.319391] usbserial: USB Serial support registered for generic [ 3.321661] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.327526] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.329227] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.331480] mousedev: PS/2 mouse device common for all mice [ 3.334319] rtc_cmos 00:05: RTC can wake from S4 [ 3.336809] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.337174] rtc_cmos 00:05: registered as rtc0 [ 3.342584] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.346550] intel_pstate: CPU model not supported [ 3.350271] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.354579] hid: raw HID events driver (C) Jiri Kosina [ 3.357376] usbcore: registered new interface driver usbhid [ 3.357473] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.359740] usbhid: USB HID core driver [ 3.364442] drop_monitor: Initializing network drop monitor service [ 3.367731] Initializing XFRM netlink socket [ 3.370433] NET: Registered protocol family 10 [ 3.373749] Segment Routing with IPv6 [ 3.375147] NET: Registered protocol family 17 [ 3.377252] mpls_gso: MPLS GSO support [ 3.383791] RAS: Correctable Errors collector initialized. [ 3.386351] AVX version of gcm_enc/dec engaged. [ 3.388411] AES CTR mode by8 optimization enabled [ 3.495291] sched_clock: Marking stable (3495067518, 0)->(4383744614, -888677096) [ 3.499487] registered taskstats version 1 [ 3.501910] Loading compiled-in X.509 certificates [ 3.506255] zswap: loaded using pool lzo/zbud [ 3.536218] Key type big_key registered [ 3.550926] Key type encrypted registered [ 3.552730] ima: No TPM chip found, activating TPM-bypass! [ 3.554830] ima: Allocated hash algorithm: sha1 [ 3.556661] ima: No architecture policies found [ 3.558434] evm: Initialising EVM extended attributes: [ 3.560517] evm: security.selinux [ 3.561814] evm: security.ima [ 3.562935] evm: security.capability [ 3.565723] evm: HMAC attrs: 0x1 [ 3.568292] rtc_cmos 00:05: setting system clock to 2026-04-14 18:23:03 UTC (1776190983) [ 3.575114] debug: unmapping init [mem 0xffffffffb8003000-0xffffffffb81fffff] [ 3.578824] debug: unmapping init [mem 0xffffffffb6d82000-0xffffffffb7058fff] [ 3.587091] Write protecting the kernel read-only data: 28672k [ 3.590651] debug: unmapping init [mem 0xffffffffb5403000-0xffffffffb55fffff] [ 3.593817] debug: unmapping init [mem 0xffffffffb5d14000-0xffffffffb5dfffff] [ 3.640934] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.650193] systemd[1]: Detected virtualization kvm. [ 3.652071] systemd[1]: Detected architecture x86-64. [ 3.654130] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.680845] systemd[1]: No hostname configured. [ 3.682939] systemd[1]: Set hostname to . [ 3.685513] random: systemd: uninitialized urandom read (16 bytes read) [ 3.687917] systemd[1]: Initializing machine ID from random generator. [ 3.753753] random: ln: uninitialized urandom read (6 bytes read) [ 3.852626] random: systemd: uninitialized urandom read (16 bytes read) [ 3.856608] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.864051] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.873205] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. [ OK ] Reached target Slices. Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.569760] device-mapper: uevent: version 1.0.3 [ 4.572135] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ 4.992140] random: fast init done Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.412082] virtio_net virtio0 ens2: renamed from eth0 [ 5.471430] scsi host0: ata_piix [ 5.480386] scsi host1: ata_piix [ 5.482060] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.484679] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.736967] dracut-initqueue[579]: RTNETLINK answers: File exists [ 10.078084] random: crng init done [ 10.079809] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 12.126984] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.424511] printk: systemd: 25 output lines suppressed due to ratelimiting [ 15.084485] SELinux: Disabled at runtime. [ 15.203979] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.227125] systemd[1]: Detected virtualization kvm. [ 15.231277] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.571833] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.575900] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.594763] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.606267] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.613169] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.627096] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.654822] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ 16.757921] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Reached target Swap. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 18.011916] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.096773] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.100914] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.505332] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.566339] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit)[ 24.283263] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 25.018162] NFS: Registering the id_resolver key type [ 25.023128] Key type id_resolver registered [ 25.033747] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg319-server login: [ 66.153869] libcfs: loading out-of-tree module taints kernel. [ 66.187419] alg: No test for adler32 (adler32-zlib) [ 66.944063] Key type ._llcrypt registered [ 66.949606] Key type .llcrypt registered [ 67.065840] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 80.016901] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 80.844051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 81.245527] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 81.728125] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 81.735764] LNet: Accept secure, port 988 [ 83.407984] Key type lgssc registered [ 84.520612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 96.560879] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 98.529958] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 106.220181] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 111.882242] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 117.692338] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 130.098405] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 140.675168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 140.774842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 141.993556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 142.017858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 142.089950] Lustre: lustre-MDT0000: new disk, initializing [ 142.177058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 142.208872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 146.578481] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 151.840278] hrtimer: interrupt took 5863259 ns [ 157.339251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 157.397068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 157.439917] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 157.457710] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 157.462061] Lustre: Skipped 1 previous similar message [ 157.559339] Lustre: lustre-MDT0001: new disk, initializing [ 157.604374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 157.638679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 157.647548] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 160.796685] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 170.312521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 170.372947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 170.380930] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 170.536143] Lustre: lustre-OST0000: new disk, initializing [ 170.540384] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 170.598479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 174.016418] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 177.164125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 177.190216] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 185.351147] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 185.402594] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 185.412599] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 185.487927] Lustre: lustre-OST0001: new disk, initializing [ 185.491786] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 185.543700] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 186.730370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 186.745837] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 189.231515] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 198.287665] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 208.127898] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 218.829499] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing check_logdir /tmp/testlogs/ [ 224.204582] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing yml_node [ 229.799509] Lustre: DEBUG MARKER: Client: 2.15.8.10 [ 232.610284] Lustre: DEBUG MARKER: MDS: 2.15.8.10 [ 235.234461] Lustre: DEBUG MARKER: OSS: 2.15.8.10 [ 236.613874] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Apr 14 14:26:55 EDT 2026 [ 244.026996] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball 110 [ 245.666943] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 263.648784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 263.664451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 263.667550] Lustre: Skipped 3 previous similar messages [ 263.685243] Lustre: Skipped 3 previous similar messages [ 267.123929] Lustre: server umount lustre-MDT0000 complete [ 268.771651] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 268.789049] LustreError: Skipped 3 previous similar messages [ 270.983427] LustreError: 6240:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776191250 with bad export cookie 4863006027975365421 [ 270.985931] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 270.996927] LustreError: 6240:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 273.888505] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 273.890366] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 273.921877] LustreError: Skipped 1 previous similar message [ 273.923811] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 275.935641] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 275.935700] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 275.945361] Lustre: Skipped 2 previous similar messages [ 275.966318] LustreError: Skipped 1 previous similar message [ 277.555158] Lustre: server umount lustre-MDT0001 complete [ 282.023459] Lustre: server umount lustre-OST0000 complete [ 286.656297] Lustre: server umount lustre-OST0001 complete [ 292.870243] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 302.650930] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 312.271624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 318.308706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 323.910751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 329.280215] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 335.742764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 335.816310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 336.036234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 336.061670] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 336.140350] Lustre: lustre-MDT0000: new disk, initializing [ 336.210827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 336.227872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 340.617702] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 349.774346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 349.855294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 349.942771] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 349.981939] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 349.985717] Lustre: Skipped 1 previous similar message [ 350.079661] Lustre: lustre-MDT0001: new disk, initializing [ 350.151419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 350.198836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 350.208922] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 354.000338] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 362.084918] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 366.762901] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 372.541891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 372.625273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 372.638232] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 372.907774] Lustre: lustre-OST0000: new disk, initializing [ 372.913647] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 372.987438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 374.073550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 374.082245] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 376.757993] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 384.949658] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 389.219817] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 389.377200] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 393.533851] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 393.705318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 399.840156] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 399.844416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 399.853688] LustreError: Skipped 1 previous similar message [ 399.864254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 399.868618] Lustre: Skipped 2 previous similar messages [ 399.873677] Lustre: Skipped 2 previous similar messages [ 401.677549] Lustre: server umount lustre-OST0000 complete [ 404.963797] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 404.968281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 404.974536] Lustre: Skipped 1 previous similar message [ 404.980570] Lustre: Skipped 2 previous similar messages [ 415.201548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 415.211655] Lustre: Skipped 3 previous similar messages [ 419.295338] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 419.393764] Lustre: server umount lustre-MDT0000 complete [ 420.325156] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 420.336616] LustreError: Skipped 3 previous similar messages [ 422.357951] LustreError: 16429:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776191402 with bad export cookie 4863006027975366870 [ 422.362563] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 422.369341] LustreError: 16429:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 425.936448] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 14:30:05 (1776191405) [ 427.947721] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 429.319839] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 14:30:08 (1776191408) [ 434.298714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 434.773403] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 434.839073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 439.333942] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 445.410201] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 445.433026] LustreError: Skipped 1 previous similar message [ 446.790176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 451.403156] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 458.744411] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 464.019695] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 470.034681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 474.728233] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 482.053874] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 496.103638] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 496.103861] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 496.109091] LustreError: Skipped 1 previous similar message [ 496.110100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 496.140182] Lustre: Skipped 1 previous similar message [ 501.054408] Lustre: server umount lustre-OST0000 complete [ 501.057301] Lustre: Skipped 1 previous similar message [ 506.338073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 506.362464] Lustre: Skipped 1 previous similar message [ 511.467410] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.030812] LustreError: 18866:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776191493 with bad export cookie 4863006027975367724 [ 514.039292] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 514.046253] LustreError: 18866:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 526.658936] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 529.048496] Key type lgssc unregistered [ 529.391717] LNet: 22561:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 529.402247] LNet: Removed LNI 192.168.203.119@tcp [ 530.156488] Key type .llcrypt unregistered [ 530.166578] Key type ._llcrypt unregistered [ 549.553450] alg: No test for adler32 (adler32-zlib) [ 550.307435] Key type ._llcrypt registered [ 550.309338] Key type .llcrypt registered [ 550.379961] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 551.161109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 551.561985] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 551.717939] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 551.720609] LNet: Accept secure, port 988 [ 553.343079] Key type lgssc registered [ 554.294088] Lustre: Echo OBD driver; http://www.lustre.org/ [ 561.680521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 563.041670] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.123717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 567.422935] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 568.303306] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.408746] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 575.506679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 575.858447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 580.382580] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 587.704963] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 593.514343] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 600.087951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 600.479369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 603.753430] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 609.868253] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 613.810369] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 624.102500] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 624.103051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 624.112459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 624.124414] Lustre: Skipped 1 previous similar message [ 628.536721] Lustre: server umount lustre-OST0000 complete [ 632.288082] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 632.304932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 632.319805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 632.330911] Lustre: Skipped 1 previous similar message [ 634.336665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 634.343236] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 637.868910] Lustre: server umount lustre-MDT0000 complete [ 639.456526] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 639.483373] LustreError: Skipped 1 previous similar message [ 641.812682] LustreError: 24006:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776191621 with bad export cookie 14528357368219609141 [ 641.822063] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 641.832527] LustreError: 24006:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 642.179954] Lustre: server umount lustre-MDT0001 complete [ 652.890636] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 654.859561] Key type lgssc unregistered [ 655.005384] LNet: 27700:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.012296] LNet: Removed LNI 192.168.203.119@tcp [ 655.797962] Key type .llcrypt unregistered [ 655.802963] Key type ._llcrypt unregistered [ 680.122499] alg: No test for adler32 (adler32-zlib) [ 680.878853] Key type ._llcrypt registered [ 680.886728] Key type .llcrypt registered [ 681.039447] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 682.237671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 683.099932] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 683.380658] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 683.383676] LNet: Accept secure, port 988 [ 685.063172] Key type lgssc registered [ 686.293556] Lustre: Echo OBD driver; http://www.lustre.org/ [ 694.165681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 695.696769] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 695.745637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 699.734736] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 700.898771] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 706.020522] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 707.505650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 707.768891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 711.947131] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 719.755845] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 724.842794] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 730.905684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 731.139209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 735.317463] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 742.250638] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 745.456495] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 753.251590] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.19@tcp (stopping) [ 754.147942] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 754.151837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 754.159116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 755.682649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 755.687031] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 755.693047] Lustre: Skipped 1 previous similar message [ 755.706640] Lustre: Skipped 1 previous similar message [ 757.772767] Lustre: server umount lustre-MDT0000 complete [ 758.369134] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 760.716492] LustreError: 29190:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776191740 with bad export cookie 346687850790118089 [ 760.728072] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 760.800408] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 760.814413] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 760.817330] Lustre: Skipped 1 previous similar message [ 763.481179] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 763.504253] LustreError: Skipped 1 previous similar message [ 765.558913] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.19@tcp (stopping) [ 765.567909] Lustre: Skipped 1 previous similar message [ 771.681127] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 771.699323] LustreError: Skipped 4 previous similar messages [ 775.135159] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 775.309573] Lustre: server umount lustre-MDT0001 complete [ 790.136120] Lustre: server umount lustre-OST0000 complete [ 807.217941] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 809.748985] Key type lgssc unregistered [ 810.033962] LNet: 33193:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.041243] LNet: Removed LNI 192.168.203.119@tcp [ 810.764186] Key type .llcrypt unregistered [ 810.766876] Key type ._llcrypt unregistered [ 831.121988] alg: No test for adler32 (adler32-zlib) [ 831.884971] Key type ._llcrypt registered [ 831.887781] Key type .llcrypt registered [ 831.997929] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 833.165778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 833.603922] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 833.866880] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 833.876266] LNet: Accept secure, port 988 [ 835.552436] Key type lgssc registered [ 836.602069] Lustre: Echo OBD driver; http://www.lustre.org/ [ 843.364455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 844.781119] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 844.824870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 848.940085] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 849.891379] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 855.015897] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 856.210419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 856.566129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 860.264300] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 867.143852] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 872.361313] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 878.718189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 879.137214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 883.347291] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 890.714254] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 894.524429] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 901.603680] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 901.617059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 901.636945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 904.678325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 904.678355] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 904.696017] Lustre: Skipped 1 previous similar message [ 904.707845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 906.152871] Lustre: server umount lustre-OST0000 complete [ 923.103952] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 923.107901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 923.115370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 923.118509] Lustre: Skipped 1 previous similar message [ 925.153311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 925.153726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 928.452711] LustreError: 37689:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x2:0x0].0x0 (00000000ab52dfe1) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 930.281711] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 930.301116] LustreError: Skipped 1 previous similar message [ 930.549693] Lustre: server umount lustre-MDT0000 complete [ 935.133559] LustreError: 34638:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776191915 with bad export cookie 6629298239822630512 [ 935.148849] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 935.149290] LustreError: 34638:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 941.667446] Lustre: server umount lustre-MDT0001 complete [ 953.296793] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 955.432479] Key type lgssc unregistered [ 955.622133] LNet: 38496:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 955.642413] LNet: Removed LNI 192.168.203.119@tcp [ 956.217424] Key type .llcrypt unregistered [ 956.222331] Key type ._llcrypt unregistered [ 976.354838] alg: No test for adler32 (adler32-zlib) [ 977.112549] Key type ._llcrypt registered [ 977.114993] Key type .llcrypt registered [ 977.254522] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 978.263477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 978.672469] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 978.902179] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 978.905588] LNet: Accept secure, port 988 [ 980.559507] Key type lgssc registered [ 981.411589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 987.647648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 989.047362] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 989.084193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 993.087714] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 993.126397] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 998.374800] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 999.865203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1000.126792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1003.784277] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 1010.628749] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1015.779626] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1021.645744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1022.012572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1025.633419] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 1034.937720] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1038.389406] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 1044.817426] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1048.254912] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1055.207837] LNetError: 120-3: Refusing connection from 192.168.203.119 for 127.0.0.2@tcp: No matching NI [ 1055.227808] LNetError: 39258:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1055.231107] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1061.367542] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-04-1414h40m38s [ 1063.777422] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1066.620446] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.19@tcp (stopping) [ 1066.976783] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1067.003272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1069.032069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1069.036996] Lustre: Skipped 1 previous similar message [ 1069.041169] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1069.066335] LNetError: 120-3: Refusing connection from 192.168.203.119 for 127.0.0.2@tcp: No matching NI [ 1069.079266] LNetError: 39256:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1069.082914] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1071.705456] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.19@tcp (stopping) [ 1071.723950] Lustre: Skipped 2 previous similar messages [ 1072.646428] Lustre: server umount lustre-MDT0000 complete [ 1074.150726] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1074.173536] LustreError: Skipped 2 previous similar messages [ 1074.174041] LNetError: 120-3: Refusing connection from 192.168.203.119 for 127.0.0.2@tcp: No matching NI [ 1074.199266] LNetError: 39259:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1074.209884] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1076.610226] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1079.266680] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1079.297055] LustreError: Skipped 3 previous similar messages [ 1079.302910] LNetError: 120-3: Refusing connection from 192.168.203.119 for 127.0.0.2@tcp: No matching NI [ 1079.323285] LNetError: 39257:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1079.330953] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1081.311267] Lustre: 39268:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776192054/real 1776192054] req@0000000011565c22 x1862472258571712/t0(0) o400->MGC192.168.203.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776192061 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1081.335375] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1083.525826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1087.457037] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1087.458617] Lustre: Evicted from MGS (at 192.168.203.119@tcp) after server handle changed from 0x4a691d078c6653cb to 0x4a691d078c665824 [ 1087.463187] LustreError: Skipped 10 previous similar messages [ 1087.469309] Lustre: MGC192.168.203.119@tcp: Connection restored to 192.168.203.119@tcp (at 0@lo) [ 1087.616290] LNetError: 120-3: Refusing connection from 192.168.203.119 for 127.0.0.2@tcp: No matching NI [ 1087.635965] LNetError: 39258:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1087.641165] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1087.654876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1087.706708] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 1090.753544] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 1093.098239] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1093.130205] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 1093.141040] Lustre: Skipped 2 previous similar messages [ 1093.420050] Lustre: DEBUG MARKER: Wait for df (28921) ... [ 1095.160770] Lustre: DEBUG MARKER: done [ 1099.744357] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1099.761082] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1099.775973] Lustre: Skipped 1 previous similar message [ 1099.787654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1103.330524] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1103.338744] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1105.292379] Lustre: server umount lustre-OST0000 complete [ 1113.569334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1113.572506] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1113.589766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1113.601304] Lustre: Skipped 3 previous similar messages [ 1113.605618] LNetError: 120-3: Refusing connection from 192.168.203.119 for 127.0.0.2@tcp: No matching NI [ 1113.610523] LNetError: 39256:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1113.627180] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1114.828219] Lustre: server umount lustre-MDT0000 complete [ 1118.107838] LustreError: 39937:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192098 with bad export cookie 5361848749618780196 [ 1118.111513] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1118.122373] LustreError: 39937:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1118.490487] Lustre: server umount lustre-MDT0001 complete [ 1130.704830] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 1133.623360] Key type lgssc unregistered [ 1133.900482] LNet: 45061:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1133.912377] LNet: Removed LNI 192.168.203.119@tcp [ 1134.636713] Key type .llcrypt unregistered [ 1134.642762] Key type ._llcrypt unregistered [ 1168.691403] alg: No test for adler32 (adler32-zlib) [ 1169.454363] Key type ._llcrypt registered [ 1169.459927] Key type .llcrypt registered [ 1169.602603] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 1184.245562] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 1185.475761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1185.810534] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 1186.021637] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 1186.027684] LNet: Accept secure, port 988 [ 1187.680021] Key type lgssc registered [ 1188.642120] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1195.639798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1201.280770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1208.688291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1216.437153] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1222.593566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1222.657796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1223.914156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1223.958440] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1224.064889] Lustre: lustre-MDT0000: new disk, initializing [ 1224.158669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1224.186532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1228.820566] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 1238.875667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1238.976154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1239.087099] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1239.164211] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1239.171036] Lustre: Skipped 1 previous similar message [ 1239.272717] Lustre: lustre-MDT0001: new disk, initializing [ 1239.354308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1239.419326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1239.425786] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1243.913959] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 1253.433114] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1258.761492] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1265.004907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1265.041594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1265.047588] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1265.212587] Lustre: lustre-OST0000: new disk, initializing [ 1265.218870] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1265.316352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1269.781937] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 1275.451755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1275.460136] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1278.530944] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1283.024523] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1283.213650] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1287.990479] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1288.198992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1290.724252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1290.728221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1290.737910] Lustre: Skipped 1 previous similar message [ 1295.843743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1295.857337] Lustre: Skipped 2 previous similar messages [ 1296.318416] Lustre: server umount lustre-OST0000 complete [ 1300.960903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1300.961107] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1300.963851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1300.980377] Lustre: Skipped 1 previous similar message [ 1306.081943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1306.093334] Lustre: Skipped 2 previous similar messages [ 1306.478054] Lustre: server umount lustre-MDT0000 complete [ 1310.008613] LustreError: 51262:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192289 with bad export cookie 15957235484558731950 [ 1310.009508] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1310.020045] LustreError: 51262:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1310.242926] Lustre: server umount lustre-MDT0001 complete [ 1318.085560] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 14:44:56 (1776192296) [ 1324.775380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1325.159874] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1325.214556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1329.547611] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 1330.658335] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1335.776534] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1337.742814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1338.058843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1342.214176] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 1349.787432] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1355.228419] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1361.961293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1362.703438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1368.427156] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 1378.033504] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1383.572921] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-04-1414h46m02s [ 1384.837342] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1391.578809] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1394.313610] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1394.376116] LustreError: 53723:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1396.740153] LNetError: 120-3: Refusing connection from 192.168.203.119 for 127.0.0.2@tcp: No matching NI [ 1396.760113] LNetError: 47391:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1396.768596] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1414.811570] Lustre: lustre-MDT0000: Client b5d9f588-496b-4f79-af72-c1373fd8f51c (at 192.168.203.19@tcp) reconnecting [ 1434.455125] LustreError: 53723:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 awake [ 1436.353341] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1438.914356] Lustre: DEBUG MARKER: done [ 1447.910326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1447.912161] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1447.938390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1447.954969] Lustre: Skipped 1 previous similar message [ 1452.486056] Lustre: server umount lustre-OST0000 complete [ 1458.146491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1458.170061] Lustre: Skipped 1 previous similar message [ 1458.196351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1458.262849] LNetError: 120-3: Refusing connection from 192.168.203.119 for 127.0.0.2@tcp: No matching NI [ 1458.277630] LNetError: 47389:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1458.298541] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1463.053025] Lustre: server umount lustre-MDT0000 complete [ 1463.267721] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1463.290078] LustreError: Skipped 1 previous similar message [ 1463.303947] LNetError: 120-3: Refusing connection from 192.168.203.119 for 127.0.0.2@tcp: No matching NI [ 1463.309745] LNetError: 47390:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1463.318814] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1466.130308] LustreError: 53710:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192446 with bad export cookie 15957235484558732804 [ 1466.134248] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1466.139577] LustreError: 53710:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1476.449690] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 1479.278892] Key type lgssc unregistered [ 1479.557596] LNet: 58144:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1479.572212] LNet: Removed LNI 192.168.203.119@tcp [ 1480.345969] Key type .llcrypt unregistered [ 1480.352287] Key type ._llcrypt unregistered [ 1512.977229] alg: No test for adler32 (adler32-zlib) [ 1513.733034] Key type ._llcrypt registered [ 1513.736412] Key type .llcrypt registered [ 1513.844947] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 1526.361910] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 1527.406737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1527.680910] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 1527.879743] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 1527.884080] LNet: Accept secure, port 988 [ 1529.551897] Key type lgssc registered [ 1530.519504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1539.012712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1546.958094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1553.393951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1560.354097] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1567.001920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1567.123555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1568.409768] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1568.437813] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1568.487465] Lustre: lustre-MDT0000: new disk, initializing [ 1568.539258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1568.563452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1573.772303] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 1586.852602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1586.976745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1587.127737] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1587.182672] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1587.187831] Lustre: Skipped 1 previous similar message [ 1587.295466] Lustre: lustre-MDT0001: new disk, initializing [ 1587.418346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1587.482667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1587.502512] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1592.457591] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 1602.756402] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1609.320650] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1614.567642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1614.610326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1614.617616] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1614.763443] Lustre: lustre-OST0000: new disk, initializing [ 1614.766804] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1614.821167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1618.944843] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 1621.564398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1621.572249] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1632.070882] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1639.626241] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1639.973225] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1645.118954] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1645.383670] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1652.704712] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1652.711443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1652.713609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1652.745539] Lustre: Skipped 1 previous similar message [ 1653.853625] Lustre: server umount lustre-OST0000 complete [ 1657.825896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1657.828045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1657.846995] Lustre: Skipped 1 previous similar message [ 1657.863573] Lustre: Skipped 1 previous similar message [ 1662.950068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1662.964436] Lustre: Skipped 2 previous similar messages [ 1663.844603] Lustre: server umount lustre-MDT0000 complete [ 1666.855491] LustreError: 62236:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192646 with bad export cookie 9914705528841798562 [ 1666.860800] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1666.867981] LustreError: 62236:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1667.076595] Lustre: server umount lustre-MDT0001 complete [ 1676.319530] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 14:50:54 (1776192654) [ 1678.249710] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1680.380295] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 14:50:58 (1776192658) [ 1692.519982] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 1696.765348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1697.132798] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1697.175663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1698.124868] Lustre: Failing over lustre-MDT0000 [ 1698.129610] LustreError: 67261:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1698.147369] LustreError: 67261:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1698.159796] LustreError: 67261:0:(lod_dev.c:528:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 1704.415176] Lustre: 67312:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776192678/real 1776192678] req@00000000960bcbf0 x1862472834242112/t0(0) o251->MGC192.168.203.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776192684 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1704.575983] Lustre: server umount lustre-MDT0000 complete [ 1711.157388] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 14:51:29 (1776192689) [ 1717.746481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1718.241822] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1718.285812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1723.053965] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 1723.368497] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1728.496031] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1730.852033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1731.147900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1735.135656] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 1741.904739] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1749.946669] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1758.725196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1759.082115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1763.959532] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 1771.918124] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1777.314327] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1787.365101] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1787.389218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1787.408911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1790.169910] Lustre: server umount lustre-MDT0000 complete [ 1793.822528] LustreError: 67862:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192773 with bad export cookie 9914705528841799689 [ 1793.827323] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1793.843188] LustreError: 67862:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1797.233512] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1800.164634] Lustre: 60464:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776192773/real 1776192773] req@000000009a26132b x1862472834261952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776192780 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1800.185155] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1800.201504] Lustre: Skipped 2 previous similar messages [ 1801.172419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1804.131138] Lustre: 60462:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776192777/real 1776192777] req@0000000038e3c493 x1862472834262016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776192784 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1806.303110] Lustre: 60461:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776192779/real 1776192779] req@000000001602bda5 x1862472834262080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776192786 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1809.333573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1810.471317] Lustre: Evicted from MGS (at 192.168.203.119@tcp) after server handle changed from 0x89981c25eb378809 to 0x89981c25eb378f9c [ 1810.482730] Lustre: MGC192.168.203.119@tcp: Connection restored to (at 0@lo) [ 1810.833562] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1810.923138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1814.909361] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 1820.137832] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1820.149108] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1820.153384] LustreError: Skipped 5 previous similar messages [ 1822.837965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1823.254740] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:65 [ 1827.203331] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 1828.330809] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1833.988602] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1840.425406] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1848.071587] Lustre: server umount lustre-MDT0000 complete [ 1848.082813] Lustre: Skipped 1 previous similar message [ 1848.806647] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1848.813395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1848.825888] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1848.840077] Lustre: Skipped 2 previous similar messages [ 1848.867743] LustreError: Skipped 2 previous similar messages [ 1853.773551] LustreError: 67862:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192833 with bad export cookie 9914705528841801628 [ 1853.774510] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1853.792398] LustreError: 67862:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1853.931288] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1853.934376] Lustre: Skipped 3 previous similar messages [ 1860.994775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1870.956501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1871.330400] Lustre: Evicted from MGS (at 192.168.203.119@tcp) after server handle changed from 0x89981c25eb378f9c to 0x89981c25eb3795ae [ 1871.340290] Lustre: MGC192.168.203.119@tcp: Connection restored to (at 0@lo) [ 1871.624130] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 1875.620276] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1876.012159] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 1881.056994] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1881.074328] LustreError: Skipped 4 previous similar messages [ 1885.481530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1885.968398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1885.975149] Lustre: Skipped 2 previous similar messages [ 1886.049997] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:97 [ 1890.453804] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 1891.315436] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1898.095840] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1903.210645] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1911.780965] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1911.783540] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1911.784084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1911.788872] Lustre: Skipped 2 previous similar messages [ 1916.585311] Lustre: server umount lustre-MDT0000 complete [ 1916.587746] Lustre: Skipped 1 previous similar message [ 1919.913413] LustreError: 69833:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192899 with bad export cookie 9914705528841803182 [ 1919.918619] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1919.921881] LustreError: 69833:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1926.911664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1928.098947] Lustre: 60462:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776192901/real 1776192901] req@000000006c673c18 x1862472834288832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776192908 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1928.126960] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1928.137721] Lustre: Skipped 1 previous similar message [ 1930.481790] Lustre: DEBUG MARKER: files compared the same [ 1952.529446] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 1955.749785] Key type lgssc unregistered [ 1956.096930] LNet: 77919:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1956.110441] LNet: Removed LNI 192.168.203.119@tcp [ 1956.986669] Key type .llcrypt unregistered [ 1956.997465] Key type ._llcrypt unregistered [ 1981.749192] alg: No test for adler32 (adler32-zlib) [ 1982.506436] Key type ._llcrypt registered [ 1982.508964] Key type .llcrypt registered [ 1982.691899] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 1983.927243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1984.809574] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 1985.085307] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 1985.090593] LNet: Accept secure, port 988 [ 1986.847116] Key type lgssc registered [ 1987.773892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1996.447227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1998.025619] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1998.102795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2002.156209] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2002.777417] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug +malloc all 8 [ 2007.524028] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2010.607628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2011.123409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2016.470516] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug +malloc all 8 [ 2026.537585] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2034.399391] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2042.353087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2042.738814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2047.477621] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug +malloc all 8 [ 2047.888035] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 2058.240476] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2061.245824] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:129 [ 2071.521572] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.523825] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2071.525692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2071.553454] Lustre: Skipped 1 previous similar message [ 2076.111546] Lustre: server umount lustre-OST0000 complete [ 2081.760178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2081.761774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2081.781664] Lustre: Skipped 1 previous similar message [ 2081.789940] Lustre: Skipped 1 previous similar message [ 2086.016742] Lustre: server umount lustre-MDT0000 complete [ 2086.880070] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2086.899654] LustreError: Skipped 1 previous similar message [ 2089.790313] LustreError: 79383:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776193069 with bad export cookie 5949519416273670787 [ 2089.793528] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2089.798098] LustreError: 79383:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2090.154560] Lustre: server umount lustre-MDT0001 complete [ 2101.494835] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 2104.493705] Key type lgssc unregistered [ 2104.782505] LNet: 83064:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2104.796135] LNet: Removed LNI 192.168.203.119@tcp [ 2105.695711] Key type .llcrypt unregistered [ 2105.700285] Key type ._llcrypt unregistered [ 2129.202947] alg: No test for adler32 (adler32-zlib) [ 2129.959224] Key type ._llcrypt registered [ 2129.971188] Key type .llcrypt registered [ 2130.031102] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 2130.935545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2131.205393] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 2131.431514] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 2131.437939] LNet: Accept secure, port 988 [ 2133.095122] Key type lgssc registered [ 2134.459868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2143.177242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2160.928383] LustreError: 84500:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.119@tcp: failed processing log, type 1: rc = -5 [ 2191.648362] LustreError: 84500:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.119@tcp: failed processing log, type 4: rc = -110 [ 2221.407171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2226.066433] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 2233.540167] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2241.081269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2241.175927] Lustre: *** cfs_fail_loc=706, val=0*** [ 2245.187401] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2245.273970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2245.341707] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 2249.500576] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 2250.725576] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2255.846717] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2258.304974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2258.660753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2258.714802] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:161 [ 2263.604788] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 2271.485471] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2277.569172] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2281.445363] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2281.455453] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2281.479336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2281.486192] Lustre: Skipped 1 previous similar message [ 2281.509587] Lustre: Skipped 1 previous similar message [ 2285.788346] Lustre: server umount lustre-OST0000 complete [ 2291.680164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2291.704269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2295.868154] Lustre: server umount lustre-MDT0000 complete [ 2296.801179] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2296.826435] LustreError: Skipped 1 previous similar message [ 2300.565505] LustreError: 84506:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776193280 with bad export cookie 18223174111424761331 [ 2300.574589] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2300.581259] LustreError: 84506:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2301.049850] Lustre: server umount lustre-MDT0001 complete [ 2312.701662] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 2315.242054] Key type lgssc unregistered [ 2315.471268] LNet: 88251:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2315.477752] LNet: Removed LNI 192.168.203.119@tcp [ 2316.253334] Key type .llcrypt unregistered [ 2316.256942] Key type ._llcrypt unregistered [ 2340.913073] alg: No test for adler32 (adler32-zlib) [ 2341.667632] Key type ._llcrypt registered [ 2341.671605] Key type .llcrypt registered [ 2341.792247] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 2342.781093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2343.187954] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 2343.394416] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 2343.397481] LNet: Accept secure, port 988 [ 2345.039203] Key type lgssc registered [ 2346.003634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2353.486203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2359.645093] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 2367.981291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2368.362490] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2368.425122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2371.939477] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 2373.602605] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2373.615036] LustreError: Skipped 2 previous similar messages [ 2378.721037] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2378.745126] LustreError: Skipped 1 previous similar message [ 2379.336634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2379.864852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2383.970046] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2384.008652] LustreError: Skipped 5 previous similar messages [ 2384.031901] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:193 [ 2385.862198] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 2389.474501] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2389.501160] LustreError: Skipped 2 previous similar messages [ 2395.317773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2395.415046] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:193 [ 2400.280391] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 2415.072812] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2415.079143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.094293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2415.584401] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2415.595091] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.608906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2420.087359] Lustre: server umount lustre-OST0000 complete [ 2426.338159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2426.344499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2426.347090] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2426.358281] Lustre: Skipped 1 previous similar message [ 2426.371968] Lustre: Skipped 3 previous similar messages [ 2430.142835] Lustre: server umount lustre-MDT0000 complete [ 2431.456634] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2431.473794] LustreError: Skipped 7 previous similar messages [ 2433.588767] Lustre: server umount lustre-MDT0001 complete [ 2443.232567] Lustre: 93123:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776193417/real 1776193417] req@0000000014778517 x1862473689872064/t0(0) o251->MGC192.168.203.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776193423 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2443.293518] Lustre: server umount MGS complete [ 2455.512056] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 15:03:53 (1776193433) [ 2485.406545] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 2493.704511] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 2503.501963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2509.925367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2516.321596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2522.692057] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2528.964155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2529.057338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2533.006718] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 2541.105642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2541.211435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2541.531045] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2541.620278] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2541.730880] Lustre: lustre-MDT0001: new disk, initializing [ 2541.844894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2541.893847] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2541.906498] LustreError: 97826:0:(lod_dev.c:528:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2546.705740] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 2557.416746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2557.485720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2557.495523] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2557.646452] Lustre: lustre-OST0000: new disk, initializing [ 2557.651270] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2557.798049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2562.269200] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 2565.741895] Lustre: 99057:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@000000008d436195 x1862473689876864/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1776193557 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 2572.349638] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2579.264455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2579.313908] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2579.462927] Lustre: lustre-MDT0000: new disk, initializing [ 2579.617878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2579.651112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2580.709621] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2580.723416] Lustre: 99678:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@000000000c2e7f02 x1862473689880448/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1776193572 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 2584.333319] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 2589.868771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2589.875809] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2593.255997] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2593.262940] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2613.728331] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2613.741509] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2613.753408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2617.832735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2617.842120] Lustre: Skipped 1 previous similar message [ 2619.443251] Lustre: server umount lustre-OST0000 complete [ 2622.947877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2622.961437] Lustre: Skipped 1 previous similar message [ 2622.972270] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2622.987708] LustreError: Skipped 1 previous similar message [ 2628.068370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2628.076347] Lustre: Skipped 4 previous similar messages [ 2628.859795] Lustre: server umount lustre-MDT0000 complete [ 2648.543175] Lustre: 100869:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776193622/real 1776193622] req@00000000c8122f4d x1862473689891520/t0(0) o251->MGC192.168.203.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776193628 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2648.618140] Lustre: server umount MGS complete [ 2648.623479] Lustre: Skipped 1 previous similar message [ 2659.067775] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 15:07:17 (1776193637) [ 2677.934973] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 2681.352331] Key type lgssc unregistered [ 2681.712806] LNet: 102279:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2681.722196] LNet: Removed LNI 192.168.203.119@tcp [ 2682.518325] Key type .llcrypt unregistered [ 2682.522822] Key type ._llcrypt unregistered [ 2700.252400] alg: No test for adler32 (adler32-zlib) [ 2701.010193] Key type ._llcrypt registered [ 2701.016155] Key type .llcrypt registered [ 2701.144946] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 2702.208056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2703.079090] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 2703.320421] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 2703.330054] LNet: Accept secure, port 988 [ 2705.047143] Key type lgssc registered [ 2706.100677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2713.153623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2714.521187] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2714.555314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2717.731268] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 2719.720474] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2724.837214] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2726.151618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2726.376278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2731.352534] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 2739.265105] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2744.984963] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2751.867300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2752.290347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2757.923487] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 2764.609660] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2767.351856] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2777.058798] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2777.069126] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2777.107117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2780.129513] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2780.141584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2780.146844] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2780.188738] Lustre: Skipped 1 previous similar message [ 2782.753753] Lustre: server umount lustre-OST0000 complete [ 2787.808836] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2787.828621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2787.842330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2790.372371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2790.373977] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2792.728869] Lustre: server umount lustre-MDT0000 complete [ 2795.489830] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2795.514952] LustreError: Skipped 1 previous similar message [ 2796.485861] LustreError: 103550:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776193776 with bad export cookie 8209120861269847828 [ 2796.486920] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2796.512299] LustreError: 103550:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2796.978911] Lustre: server umount lustre-MDT0001 complete [ 2808.496471] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 2811.204853] Key type lgssc unregistered [ 2811.463571] LNet: 107223:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2811.497132] LNet: Removed LNI 192.168.203.119@tcp [ 2812.280387] Key type .llcrypt unregistered [ 2812.282684] Key type ._llcrypt unregistered [ 2826.581259] alg: No test for adler32 (adler32-zlib) [ 2827.345292] Key type ._llcrypt registered [ 2827.349835] Key type .llcrypt registered [ 2827.539257] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 2828.639120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2829.020764] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 2829.242173] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 2829.249134] LNet: Accept secure, port 988 [ 2830.895290] Key type lgssc registered [ 2831.891128] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2837.604534] LustreError: 108350:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 2837.617475] LustreError: 108352:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 2837.628508] LustreError: 108350:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4987 [ 2837.641250] LustreError: 108350:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2837.649409] LustreError: 108350:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2837.652594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2839.153638] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2839.208329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2844.642829] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2846.257491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2846.568804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2850.859955] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 2855.827668] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing lsmod [ 2858.330436] LustreError: 109657:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 2858.350557] LustreError: 109660:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 2858.358777] LustreError: 109657:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4976 [ 2858.363218] LustreError: 109657:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2858.369112] LustreError: 109657:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2858.376127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2858.724493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2861.776733] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2861.786025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2861.793093] Lustre: Skipped 2 previous similar messages [ 2861.811187] Lustre: Skipped 2 previous similar messages [ 2867.171614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2867.179061] Lustre: Skipped 2 previous similar messages [ 2867.852767] Lustre: server umount lustre-MDT0000 complete [ 2871.187131] LustreError: 108371:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776193851 with bad export cookie 7974013162780140986 [ 2871.196443] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2871.208957] LustreError: 108371:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2871.447764] Lustre: server umount lustre-MDT0001 complete [ 2880.993729] Lustre: 110252:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776193854/real 1776193854] req@000000009b41da04 x1862474199475712/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776193860 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2881.119846] Lustre: server umount lustre-OST0000 complete [ 2892.303865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2893.076125] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2893.201868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2898.402233] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2899.048763] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 2903.523628] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2909.474774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2909.841119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2914.026553] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 2920.897648] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2926.919718] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2932.504891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2932.736451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2935.914388] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 2942.215913] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2943.985631] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 2954.213704] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2954.224449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2954.225942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2954.233796] Lustre: Skipped 1 previous similar message [ 2954.250041] Lustre: Skipped 1 previous similar message [ 2954.399157] Lustre: server umount lustre-OST0000 complete [ 2959.331331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2959.349989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2964.238327] Lustre: server umount lustre-MDT0000 complete [ 2964.453097] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2964.472631] LustreError: Skipped 2 previous similar messages [ 2967.763715] LustreError: 110709:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776193947 with bad export cookie 7974013162780141700 [ 2967.765914] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2967.775611] LustreError: 110709:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2979.944920] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 2982.802408] Key type lgssc unregistered [ 2983.038788] LNet: 114382:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2983.046843] LNet: Removed LNI 192.168.203.119@tcp [ 2983.884472] Key type .llcrypt unregistered [ 2983.893076] Key type ._llcrypt unregistered [ 3003.817099] alg: No test for adler32 (adler32-zlib) [ 3004.576447] Key type ._llcrypt registered [ 3004.578638] Key type .llcrypt registered [ 3004.712528] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 3005.392468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3005.792596] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 3005.964148] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 3005.971967] LNet: Accept secure, port 988 [ 3007.623190] Key type lgssc registered [ 3008.609553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3015.770861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3017.268803] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3017.371471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3021.940694] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 3022.819416] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3027.950232] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3030.347023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3030.754245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3035.379330] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 3042.531440] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3047.913171] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3054.525570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3054.904117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3059.929498] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 3067.192437] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 3068.186850] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3073.636247] Lustre: Setting parameter lustre-client.llite.some_wrong_param in log lustre-client [ 3077.487186] Lustre: Modifying parameter lustre-client.llite.some_wrong_param in log lustre-client [ 3082.724351] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3082.745813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3082.757308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3082.792535] Lustre: Skipped 1 previous similar message [ 3087.843495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3087.869973] Lustre: Skipped 1 previous similar message [ 3088.653926] Lustre: server umount lustre-OST0000 complete [ 3092.966257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3092.985450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3092.994084] Lustre: Skipped 1 previous similar message [ 3097.568265] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3097.580918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3097.603340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3103.202441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3103.214483] Lustre: Skipped 3 previous similar messages [ 3107.295129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3107.419075] Lustre: server umount lustre-MDT0000 complete [ 3108.338581] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3108.349940] LustreError: Skipped 1 previous similar message [ 3111.436361] LustreError: 115836:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776194091 with bad export cookie 15193276539950600451 [ 3111.443614] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3111.451066] LustreError: 115836:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3111.766421] Lustre: server umount lustre-MDT0001 complete [ 3123.459444] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 3125.896623] Key type lgssc unregistered [ 3126.202151] LNet: 119616:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3126.212593] LNet: Removed LNI 192.168.203.119@tcp [ 3126.884217] Key type .llcrypt unregistered [ 3126.885981] Key type ._llcrypt unregistered [ 3145.603899] alg: No test for adler32 (adler32-zlib) [ 3146.359606] Key type ._llcrypt registered [ 3146.361701] Key type .llcrypt registered [ 3146.463929] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 3147.753087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3148.182679] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 3148.330878] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 3148.338022] LNet: Accept secure, port 988 [ 3149.959481] Key type lgssc registered [ 3151.179957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3158.009193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3159.367252] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3159.409390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3163.412737] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 3164.647510] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3169.761216] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3171.114794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3171.416839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3175.621878] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 3182.582795] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3188.662727] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3195.325554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3195.574912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3199.355705] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 3206.981725] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3209.925959] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 3220.458458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3220.466370] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3220.470730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3220.472083] Lustre: Skipped 1 previous similar message [ 3225.569363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3225.575438] Lustre: Skipped 2 previous similar messages [ 3225.772224] Lustre: server umount lustre-OST0000 complete [ 3230.692696] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3230.695516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3230.719185] Lustre: Skipped 1 previous similar message [ 3235.025955] Lustre: server umount lustre-MDT0000 complete [ 3235.809445] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3235.834414] LustreError: Skipped 1 previous similar message [ 3238.465697] LustreError: 120886:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776194218 with bad export cookie 11163873181685258329 [ 3238.474719] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3238.478796] LustreError: 120886:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3238.826230] Lustre: server umount lustre-MDT0001 complete [ 3249.961049] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 3252.350566] Key type lgssc unregistered [ 3252.597644] LNet: 124588:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3252.604356] LNet: Removed LNI 192.168.203.119@tcp [ 3253.297274] Key type .llcrypt unregistered [ 3253.299420] Key type ._llcrypt unregistered [ 3273.309551] alg: No test for adler32 (adler32-zlib) [ 3274.066567] Key type ._llcrypt registered [ 3274.069061] Key type .llcrypt registered [ 3274.154051] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 3274.938732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3275.531968] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 3275.739385] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 3275.743523] LNet: Accept secure, port 988 [ 3277.415130] Key type lgssc registered [ 3278.316853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3285.707600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3287.137126] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3287.181443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3291.726644] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 3292.641675] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3297.763053] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3299.229723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3299.486535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3302.929842] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 3311.125437] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3318.855753] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3327.103434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3327.705717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3331.941556] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 3338.002632] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 3339.848175] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3346.824876] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 3350.981796] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 3350.988490] Lustre: Skipped 2 previous similar messages [ 3354.232316] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3354.280204] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3355.245962] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 3355.256498] Lustre: Skipped 2 previous similar messages [ 3364.419246] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3364.428031] Lustre: Skipped 1 previous similar message [ 3369.671663] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 3369.676294] Lustre: Skipped 2 previous similar messages [ 3374.705977] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.19@tcp 192.168.0.[2,10]@tcp [ 3382.755425] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3382.769430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3382.797150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3384.288887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3384.299165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3387.413319] Lustre: server umount lustre-OST0000 complete [ 3391.971389] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3391.985315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3391.998898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3392.002264] Lustre: Skipped 1 previous similar message [ 3394.529515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3394.529781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3394.541811] Lustre: Skipped 1 previous similar message [ 3399.650286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3405.279393] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3405.439725] Lustre: server umount lustre-MDT0000 complete [ 3405.796300] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3405.820170] LustreError: Skipped 1 previous similar message [ 3409.701921] LustreError: 126839:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776194389 with bad export cookie 17597066914053784212 [ 3409.707420] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3409.720843] LustreError: 126839:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3410.091057] Lustre: server umount lustre-MDT0001 complete [ 3422.902279] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 3425.769535] Key type lgssc unregistered [ 3426.006874] LNet: 130915:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3426.032460] LNet: Removed LNI 192.168.203.119@tcp [ 3426.781281] Key type .llcrypt unregistered [ 3426.788538] Key type ._llcrypt unregistered [ 3452.897232] alg: No test for adler32 (adler32-zlib) [ 3453.649745] Key type ._llcrypt registered [ 3453.654780] Key type .llcrypt registered [ 3453.775719] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 3454.982478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3455.435520] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 3455.651961] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 3455.665196] LNet: Accept secure, port 988 [ 3457.367143] Key type lgssc registered [ 3458.637939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3466.262578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3467.650441] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3467.683669] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3467.719730] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.203.19@tcp 192.168.0.[2,10]@tcp [ 3467.729449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3472.470099] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 3472.871758] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3477.988571] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3479.989272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3480.312074] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3480.336746] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.19@tcp 192.168.0.[2,10]@tcp [ 3480.397072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3485.149649] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 3493.837873] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3499.653728] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3506.021649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3506.358152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3510.880329] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 3517.620411] Lustre: lustre-OST0000: deleting orphan objects from 0x0:167 to 0x0:193 [ 3518.158732] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3528.169500] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3528.170469] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3528.175390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3528.201701] Lustre: Skipped 1 previous similar message [ 3533.280348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3533.286872] Lustre: Skipped 2 previous similar messages [ 3533.503137] Lustre: server umount lustre-OST0000 complete [ 3538.400374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3538.405794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3538.422738] Lustre: Skipped 1 previous similar message [ 3543.237696] Lustre: server umount lustre-MDT0000 complete [ 3543.520873] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3543.540473] LustreError: Skipped 1 previous similar message [ 3546.282072] LustreError: 132505:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776194526 with bad export cookie 13926468176776396847 [ 3546.290293] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3546.294354] LustreError: 132505:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3546.613395] Lustre: server umount lustre-MDT0001 complete [ 3559.623105] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 3562.545653] Key type lgssc unregistered [ 3562.784550] LNet: 136234:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3562.799357] LNet: Removed LNI 192.168.203.119@tcp [ 3563.614545] Key type .llcrypt unregistered [ 3563.620598] Key type ._llcrypt unregistered [ 3588.853685] alg: No test for adler32 (adler32-zlib) [ 3589.609588] Key type ._llcrypt registered [ 3589.613288] Key type .llcrypt registered [ 3589.730267] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 3602.734439] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 3603.582634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3604.157928] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 3604.359084] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 3604.363980] LNet: Accept secure, port 988 [ 3606.071206] Key type lgssc registered [ 3607.097903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3615.654156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3623.078623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3629.126503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3635.382206] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3641.849725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3641.907249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3643.134447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3643.167864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3643.264812] Lustre: lustre-MDT0000: new disk, initializing [ 3643.349513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3643.376619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3647.900990] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 3656.751196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3656.824502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3656.905703] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3656.970392] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3656.979185] Lustre: Skipped 1 previous similar message [ 3657.056551] Lustre: lustre-MDT0001: new disk, initializing [ 3657.162429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3657.217562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3657.229575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3661.141539] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 3671.056424] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3677.681366] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3684.424807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3684.513225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3684.519841] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3684.747640] Lustre: lustre-OST0000: new disk, initializing [ 3684.754128] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3684.827703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3689.819416] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 3692.079878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3692.096207] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3699.180314] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3703.149402] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3703.399267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3708.752427] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3708.945535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3712.992069] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3712.996897] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3713.007752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3713.027499] Lustre: Skipped 1 previous similar message [ 3718.126226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3718.137345] Lustre: Skipped 2 previous similar messages [ 3723.264417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3726.303127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3726.571342] Lustre: server umount lustre-OST0000 complete [ 3733.473184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3733.476332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3733.491322] Lustre: Skipped 1 previous similar message [ 3733.500544] Lustre: Skipped 2 previous similar messages [ 3736.555541] Lustre: server umount lustre-MDT0000 complete [ 3738.592135] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3738.613490] LustreError: Skipped 1 previous similar message [ 3739.953410] LustreError: 139753:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776194719 with bad export cookie 11990011044040257231 [ 3739.961768] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3739.964621] LustreError: 139753:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3740.171635] Lustre: server umount lustre-MDT0001 complete [ 3747.798462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3748.400339] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3748.491722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3752.997855] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 3753.964460] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3759.072656] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3761.511285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3761.903197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3765.778764] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 3773.794838] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3779.797199] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3786.787367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3787.244198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3791.299666] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 3798.630703] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3805.074171] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3805.403786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3811.240677] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3811.570810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3820.827181] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3820.885584] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3820.895342] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3820.998525] Lustre: lustre-OST0001: new disk, initializing [ 3821.006048] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3821.085529] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3825.251670] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 3829.288607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3829.299135] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3833.752789] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3834.036689] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3838.865195] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 3839.118676] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3844.901325] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9673c61d7000.ost_server_uuid 40 [ 3846.458587] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9673c61d7000.ost_server_uuid in IDLE state after 0 sec [ 3854.817991] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3854.826440] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3854.832068] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3854.837503] Lustre: Skipped 1 previous similar message [ 3865.064659] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3865.072463] Lustre: Skipped 4 previous similar messages [ 3869.151148] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3869.207262] Lustre: server umount lustre-OST0001 complete [ 3870.182946] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3870.210357] LustreError: Skipped 1 previous similar message [ 3873.258099] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3873.279617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3875.296400] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3878.715427] Lustre: server umount lustre-OST0000 complete [ 3885.026498] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3885.032958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3885.047364] Lustre: Skipped 1 previous similar message [ 3885.052770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3885.064403] Lustre: Skipped 3 previous similar messages [ 3888.951434] Lustre: server umount lustre-MDT0000 complete [ 3890.656232] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3890.665619] LustreError: Skipped 3 previous similar messages [ 3892.437330] LustreError: 147654:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776194872 with bad export cookie 11990011044040258085 [ 3892.447787] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3892.454799] LustreError: 147654:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3916.348373] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 3919.173581] Key type lgssc unregistered [ 3919.531265] LNet: 150767:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3919.547595] LNet: Removed LNI 192.168.203.119@tcp [ 3920.437763] Key type .llcrypt unregistered [ 3920.440639] Key type ._llcrypt unregistered [ 3953.305301] alg: No test for adler32 (adler32-zlib) [ 3954.062269] Key type ._llcrypt registered [ 3954.067809] Key type .llcrypt registered [ 3954.224516] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 3968.815893] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 3969.956102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3970.463948] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 3970.730974] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 3970.735570] LNet: Accept secure, port 988 [ 3972.407125] Key type lgssc registered [ 3973.424808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3980.825661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3987.443584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3992.821746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3999.167572] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4005.873287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4005.992363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4007.147216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4007.178593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4007.260338] Lustre: lustre-MDT0000: new disk, initializing [ 4007.372376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4007.395548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4012.642264] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 4025.479936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4025.610968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4025.761705] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4025.862560] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4025.871126] Lustre: Skipped 1 previous similar message [ 4025.975840] Lustre: lustre-MDT0001: new disk, initializing [ 4026.106716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4026.164849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4026.175409] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4030.858199] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 4042.561922] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4049.431168] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4056.389088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4056.444783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4056.450030] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4056.677988] Lustre: lustre-OST0000: new disk, initializing [ 4056.682475] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4056.789983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4060.874780] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 4064.301165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4064.316621] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4069.847477] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4074.130898] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4074.311592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4078.462200] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4078.656541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4085.215896] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4085.229736] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4085.234752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4085.247650] Lustre: Skipped 1 previous similar message [ 4085.259614] Lustre: Skipped 1 previous similar message [ 4086.507910] Lustre: server umount lustre-OST0000 complete [ 4090.338695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4090.349971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4090.355847] Lustre: Skipped 1 previous similar message [ 4095.459495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4095.467057] Lustre: Skipped 2 previous similar messages [ 4096.159082] Lustre: server umount lustre-MDT0000 complete [ 4099.076485] LustreError: 154877:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776195079 with bad export cookie 11480452129674684063 [ 4099.080316] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4099.098123] LustreError: 154877:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4099.387252] Lustre: server umount lustre-MDT0001 complete [ 4107.884151] Lustre: DEBUG MARKER: == conf-sanity test 47: server restart does not make client loss lru_resize settings ========================================================== 15:31:26 (1776195086) [ 4125.694750] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 4135.159302] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 4146.368628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4152.369678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4158.384987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4165.315973] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4172.794303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4172.851924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4173.028456] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4173.055803] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4173.130540] Lustre: lustre-MDT0000: new disk, initializing [ 4173.190767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4173.204339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4177.076608] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 4187.875663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4187.957138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4188.085305] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4188.121883] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4188.126378] Lustre: Skipped 1 previous similar message [ 4188.215288] Lustre: lustre-MDT0001: new disk, initializing [ 4188.292217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4188.322772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4188.335344] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4192.756789] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 4202.337405] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4207.220933] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4212.888220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4212.953387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4212.956386] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4213.301733] Lustre: lustre-OST0000: new disk, initializing [ 4213.310496] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4213.432178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4218.380256] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 4222.021650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4222.028361] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4227.398695] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4232.308192] Lustre: Failing over lustre-OST0000 [ 4232.444129] Lustre: server umount lustre-OST0000 complete [ 4239.840818] Lustre: 153104:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776195212/real 1776195212] req@00000000d1c7c5de x1862475395925440/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1776195219 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 4239.840913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4239.863753] Lustre: 153104:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4240.928632] Lustre: 153105:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776195213/real 1776195213] req@00000000e4e4a9b3 x1862475395925696/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1776195220 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 4244.959324] Lustre: 153104:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776195217/real 1776195217] req@00000000d9a5081b x1862475395926144/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776195224 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4244.988152] Lustre: 153104:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4247.007757] Lustre: 153105:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776195219/real 1776195219] req@00000000c84717b7 x1862475395926720/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776195226 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4251.121117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4251.466031] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4251.483886] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4255.341874] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 4256.560671] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4257.907443] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4257.911091] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 4257.924166] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 4258.473154] Lustre: Failing over lustre-MDT0000 [ 4258.640252] Lustre: server umount lustre-MDT0000 complete [ 4260.320226] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4260.333591] LustreError: Skipped 1 previous similar message [ 4260.339838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4260.352463] Lustre: Skipped 1 previous similar message [ 4260.363688] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4261.863149] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4261.882702] LustreError: Skipped 2 previous similar messages [ 4263.026171] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4266.976937] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4266.990483] LustreError: Skipped 2 previous similar messages [ 4272.096473] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4272.107664] LustreError: Skipped 3 previous similar messages [ 4277.167228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4277.344096] LustreError: 11-0: MGC192.168.203.119@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 4277.351984] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4277.361699] Lustre: Evicted from MGS (at 192.168.203.119@tcp) after server handle changed from 0x9f52c262ca22e9ee to 0x9f52c262ca22ef9e [ 4277.377840] Lustre: MGC192.168.203.119@tcp: Connection restored to (at 0@lo) [ 4277.387264] Lustre: Skipped 1 previous similar message [ 4277.667481] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4278.362682] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4282.366298] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 4282.856527] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 4282.892320] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4282.968061] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 4293.093358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4293.105678] Lustre: Skipped 2 previous similar messages [ 4293.109315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4293.114872] Lustre: Skipped 1 previous similar message [ 4297.015177] Lustre: server umount lustre-OST0000 complete [ 4303.331520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4303.336351] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4303.346524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4303.347595] Lustre: Skipped 2 previous similar messages [ 4308.450493] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4308.464172] LustreError: Skipped 5 previous similar messages [ 4310.326129] LustreError: 161867:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776195290 with bad export cookie 11480452129674686366 [ 4310.335722] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4310.339181] LustreError: 161867:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4323.342889] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 4326.540575] Key type lgssc unregistered [ 4326.729347] LNet: 167730:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4326.739875] LNet: Removed LNI 192.168.203.119@tcp [ 4327.336918] Key type .llcrypt unregistered [ 4327.339213] Key type ._llcrypt unregistered [ 4345.892062] alg: No test for adler32 (adler32-zlib) [ 4346.645447] Key type ._llcrypt registered [ 4346.650211] Key type .llcrypt registered [ 4346.768158] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 4347.490127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4347.854395] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 4348.055768] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 4348.061650] LNet: Accept secure, port 988 [ 4349.727176] Key type lgssc registered [ 4350.668331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4358.270393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4359.772844] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4359.818768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4364.058215] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 4365.285224] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4370.410372] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4371.265499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4371.629616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4375.060744] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 4381.497667] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4386.943594] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4392.713942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4393.043519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4396.972750] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 4405.173144] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4410.500518] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:97 [ 4447.362642] Lustre: lustre-MDT0000: Client 1ada4801-7866-46ee-a16f-fb6f156fb16c (at 192.168.203.19@tcp) reconnecting [ 4464.796907] Lustre: lustre-MDT0000: Client 1ada4801-7866-46ee-a16f-fb6f156fb16c (at 192.168.203.19@tcp) reconnecting [ 4483.737733] Lustre: lustre-MDT0000: Client 1ada4801-7866-46ee-a16f-fb6f156fb16c (at 192.168.203.19@tcp) reconnecting [ 4510.317834] Lustre: lustre-MDT0000: Client 1ada4801-7866-46ee-a16f-fb6f156fb16c (at 192.168.203.19@tcp) reconnecting [ 4527.740608] Lustre: lustre-MDT0000: Client 1ada4801-7866-46ee-a16f-fb6f156fb16c (at 192.168.203.19@tcp) reconnecting [ 4574.852101] Lustre: lustre-MDT0000: Client 1ada4801-7866-46ee-a16f-fb6f156fb16c (at 192.168.203.19@tcp) reconnecting [ 4607.613974] Lustre: lustre-MDT0000: Client 1ada4801-7866-46ee-a16f-fb6f156fb16c (at 192.168.203.19@tcp) reconnecting [ 4745.336276] Lustre: lustre-MDT0000: Client 1ada4801-7866-46ee-a16f-fb6f156fb16c (at 192.168.203.19@tcp) reconnecting [ 4850.864526] Lustre: lustre-MDT0000: Client 1ada4801-7866-46ee-a16f-fb6f156fb16c (at 192.168.203.19@tcp) reconnecting [ 4850.877363] Lustre: Skipped 1 previous similar message [ 4957.693823] Lustre: Failing over lustre-MDT0000 [ 4957.855787] Lustre: server umount lustre-MDT0000 complete [ 4958.323678] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4958.689776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4958.701450] Lustre: Skipped 2 previous similar messages [ 4963.414412] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4963.438141] LustreError: Skipped 3 previous similar messages [ 4965.855219] Lustre: 168503:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776195938/real 1776195938] req@000000001c03013d x1862475791590080/t0(0) o400->MGC192.168.203.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776195945 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4965.883904] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4971.616892] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4971.641422] LustreError: Skipped 11 previous similar messages [ 4977.474783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4982.324418] Lustre: Evicted from MGS (at 192.168.203.119@tcp) after server handle changed from 0x66299f9cd78b5d2c to 0x66299f9cd78dc95b [ 4982.349688] Lustre: MGC192.168.203.119@tcp: Connection restored to 192.168.203.119@tcp (at 0@lo) [ 4982.615617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4982.679912] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4986.967261] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4987.586606] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 4987.888575] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 4987.914840] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4987.958849] Lustre: lustre-OST0000: deleting orphan objects from 0x0:100 to 0x0:129 [ 4995.676791] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4997.085592] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5018.592438] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5018.598677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5018.601178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5018.633130] Lustre: Skipped 2 previous similar messages [ 5018.652439] Lustre: Skipped 1 previous similar message [ 5020.004910] Lustre: server umount lustre-MDT0000 complete [ 5023.725028] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5023.756654] LustreError: Skipped 19 previous similar messages [ 5025.476134] LustreError: 169174:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196005 with bad export cookie 7361590561893894491 [ 5025.482432] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5025.494723] LustreError: 169174:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5028.834139] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5028.847070] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5028.851849] Lustre: Skipped 1 previous similar message [ 5030.882919] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5031.980318] Lustre: server umount lustre-MDT0001 complete [ 5041.416934] Lustre: server umount lustre-OST0000 complete [ 5051.799181] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 5060.804314] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 5070.660152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5077.252807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5083.091786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5089.770208] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5096.103448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5096.182885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5096.440416] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5096.501285] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5096.606300] Lustre: lustre-MDT0000: new disk, initializing [ 5096.733334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5096.769572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5102.216724] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 5113.069306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5113.148663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5113.216450] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5113.246844] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5113.250950] Lustre: Skipped 1 previous similar message [ 5113.316815] Lustre: lustre-MDT0001: new disk, initializing [ 5113.420422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5113.459974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5113.473395] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5117.801605] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 5126.947891] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5132.955680] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5139.695423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5139.746660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5139.750773] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5139.997498] Lustre: lustre-OST0000: new disk, initializing [ 5140.002915] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5140.081751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5143.982690] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 5147.713920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5147.723747] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5152.893835] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5158.503334] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5158.689450] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5163.273881] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5163.565244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5168.614204] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5168.618358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5168.619753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5168.627827] LustreError: Skipped 1 previous similar message [ 5168.664500] Lustre: Skipped 1 previous similar message [ 5171.493321] Lustre: server umount lustre-OST0000 complete [ 5178.861423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5178.865829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5178.879224] Lustre: Skipped 1 previous similar message [ 5178.893777] Lustre: Skipped 2 previous similar messages [ 5181.520101] Lustre: server umount lustre-MDT0000 complete [ 5183.969905] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5183.984338] LustreError: Skipped 3 previous similar messages [ 5185.369358] LustreError: 176129:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196165 with bad export cookie 7361590561893895205 [ 5185.379840] LustreError: 176129:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5185.387732] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5193.545984] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:49:32 (1776196172) [ 5208.946760] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 5217.026789] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 5226.674631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5232.916351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5238.682126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5244.549397] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5251.278219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5251.335829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5251.547360] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5251.577798] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5251.662550] Lustre: lustre-MDT0000: new disk, initializing [ 5251.805219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5251.834392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5256.243828] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 5265.777690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5265.879865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5265.972896] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5266.018485] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5266.022403] Lustre: Skipped 1 previous similar message [ 5266.082960] Lustre: lustre-MDT0001: new disk, initializing [ 5266.213017] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5266.221708] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5270.276469] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 5279.274103] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5284.952387] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5291.006775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5291.061916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5291.070670] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5291.282221] Lustre: lustre-OST0000: new disk, initializing [ 5291.286317] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5295.854553] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 5299.276306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5299.291084] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5305.426626] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5315.041587] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5315.041670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5315.049944] LustreError: Skipped 1 previous similar message [ 5315.060762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5315.078049] Lustre: Skipped 1 previous similar message [ 5318.798324] Lustre: server umount lustre-OST0000 complete [ 5318.800432] Lustre: Skipped 1 previous similar message [ 5322.720940] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5330.404193] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5330.426284] LustreError: Skipped 3 previous similar messages [ 5333.721316] LustreError: 183116:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196313 with bad export cookie 7361590561893896059 [ 5333.726415] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5333.744019] LustreError: 183116:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5342.360626] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:52:00 (1776196320) [ 5359.599536] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 5369.647452] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 5380.285264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5387.367149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5393.521659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5399.921495] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5406.157723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5406.238786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5406.476368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5406.553188] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5406.687843] Lustre: lustre-MDT0000: new disk, initializing [ 5406.790138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5406.792762] Lustre: Skipped 2 previous similar messages [ 5406.840286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5411.124917] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 5420.337785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5420.404482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5420.494427] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5420.756275] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5424.552690] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 5433.763281] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5439.736507] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5446.440566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5446.514332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5446.526721] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5446.790531] Lustre: lustre-OST0000: new disk, initializing [ 5446.794019] Lustre: Skipped 1 previous similar message [ 5446.797223] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5446.803831] Lustre: Skipped 2 previous similar messages [ 5450.872441] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 5452.360210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5452.378765] Lustre: Skipped 1 previous similar message [ 5452.390871] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5459.856618] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5468.135117] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5468.136066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5468.141494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5468.141503] Lustre: Skipped 4 previous similar messages [ 5468.154015] LustreError: Skipped 1 previous similar message [ 5468.166257] Lustre: Skipped 3 previous similar messages [ 5473.567318] Lustre: server umount lustre-OST0000 complete [ 5473.576310] Lustre: Skipped 2 previous similar messages [ 5487.929863] LustreError: 192743:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196467 with bad export cookie 7361590561893897144 [ 5487.937106] LustreError: 192743:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5487.942703] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5500.183463] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 5502.858518] Key type lgssc unregistered [ 5503.079474] LNet: 193548:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5503.089872] LNet: Removed LNI 192.168.203.119@tcp [ 5503.857826] Key type .llcrypt unregistered [ 5503.862205] Key type ._llcrypt unregistered [ 5524.719952] alg: No test for adler32 (adler32-zlib) [ 5525.473606] Key type ._llcrypt registered [ 5525.476366] Key type .llcrypt registered [ 5525.590647] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 5526.427434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5526.691160] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 5526.816025] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 5526.823176] LNet: Accept secure, port 988 [ 5528.479121] Key type lgssc registered [ 5529.927849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5537.977610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5539.408773] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5539.483736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5544.933257] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5545.063775] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 5550.050648] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5552.895783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5553.266325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5556.623274] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 5563.084735] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5568.914943] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5575.004544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5575.309419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5579.701566] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 5586.272318] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5589.675720] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 5609.446080] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5609.454447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5609.467558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5610.467612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5610.472775] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5610.478852] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5613.366557] Lustre: server umount lustre-OST0000 complete [ 5619.680296] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5619.693112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5619.713283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5619.719187] Lustre: Skipped 1 previous similar message [ 5623.259414] Lustre: server umount lustre-MDT0000 complete [ 5625.841382] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5625.852300] LustreError: Skipped 1 previous similar message [ 5627.053025] LustreError: 194991:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196606 with bad export cookie 11716475794851098943 [ 5627.077241] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5627.082806] LustreError: 194991:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5627.492044] Lustre: server umount lustre-MDT0001 complete [ 5639.798713] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 5642.455564] Key type lgssc unregistered [ 5642.875575] LNet: 198693:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5642.898783] LNet: Removed LNI 192.168.203.119@tcp [ 5643.768360] Key type .llcrypt unregistered [ 5643.770737] Key type ._llcrypt unregistered [ 5665.889117] alg: No test for adler32 (adler32-zlib) [ 5666.637475] Key type ._llcrypt registered [ 5666.640054] Key type .llcrypt registered [ 5666.725924] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 5667.423729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5667.942445] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 5668.194018] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 5668.205054] LNet: Accept secure, port 988 [ 5669.895504] Key type lgssc registered [ 5671.125898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5679.694891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5681.255163] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5681.312194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5685.903097] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 5686.759944] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5691.875901] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5694.192667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5694.634560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5698.382752] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 5704.875215] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5710.912239] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5716.762912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5717.004934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5721.210326] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 5729.071308] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5731.384089] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 5739.488101] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5739.500205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5739.517021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5740.157130] Lustre: lustre-OST0000: Not available for connect from 192.168.203.19@tcp (stopping) [ 5741.541453] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5741.544279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5741.558617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5745.244382] Lustre: lustre-OST0000: Not available for connect from 192.168.203.19@tcp (stopping) [ 5745.255983] Lustre: Skipped 1 previous similar message [ 5745.593611] Lustre: server umount lustre-OST0000 complete [ 5752.476900] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9673c453b800.ost_server_uuid 40 [ 5754.456978] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5768.161295] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5768.162644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5768.183546] Lustre: Skipped 1 previous similar message [ 5774.285499] Lustre: server umount lustre-MDT0000 complete [ 5777.344502] LustreError: 200135:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196757 with bad export cookie 15060872059675986968 [ 5777.346298] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5777.357523] LustreError: 200135:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5777.658209] Lustre: server umount lustre-MDT0001 complete [ 5785.333905] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 15:59:24 (1776196764) [ 5791.002356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5791.377815] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5791.451786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5794.860051] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 5796.835831] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5801.714067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5801.999262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5805.594236] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 5812.896373] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5817.735952] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5823.038492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5823.279577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5826.797314] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 5833.985436] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5840.368836] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5840.445922] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5840.455994] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 5840.553959] Lustre: lustre-OST0001: new disk, initializing [ 5840.558556] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5840.618181] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5841.651872] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 5844.814830] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 5850.168099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5850.175566] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5853.566747] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5857.250941] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5857.252366] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5857.255176] LustreError: Skipped 1 previous similar message [ 5857.256912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5857.264112] Lustre: Skipped 1 previous similar message [ 5857.288643] Lustre: Skipped 4 previous similar messages [ 5862.213069] Lustre: server umount lustre-OST0000 complete [ 5864.534965] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5867.447984] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5867.591358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5871.508388] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5871.658375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5881.317740] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5881.344236] LustreError: Skipped 10 previous similar messages [ 5886.441985] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5886.445858] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5886.449451] LustreError: Skipped 1 previous similar message [ 5886.450098] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5886.450101] Lustre: Skipped 3 previous similar messages [ 5886.464770] Lustre: Skipped 1 previous similar message [ 5892.454397] Lustre: server umount lustre-OST0001 complete [ 5899.239226] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5899.250638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5903.305578] Lustre: server umount lustre-MDT0000 complete [ 5907.624951] LustreError: 204121:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196887 with bad export cookie 15060872059675988032 [ 5907.642613] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5907.645246] LustreError: 204121:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5916.082853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5920.645218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5924.696147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5928.659634] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5936.046479] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 16:01:54 (1776196914) [ 5942.301363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5942.628817] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5942.682168] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5942.966725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5947.685968] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 5956.558304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5956.635910] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5956.674075] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5960.948290] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 5969.041485] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5974.962434] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5982.146079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5982.487613] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5987.047165] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 5990.005145] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:129 [ 5993.987405] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6000.386760] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6000.471764] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6004.959639] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 6008.430406] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3 to 0x0:33 [ 6011.811421] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6015.457518] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6015.478665] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6015.490492] Lustre: Skipped 1 previous similar message [ 6015.493480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6015.501560] Lustre: Skipped 6 previous similar messages [ 6020.690501] Lustre: server umount lustre-OST0000 complete [ 6020.698836] Lustre: Skipped 1 previous similar message [ 6022.755201] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6022.777414] LustreError: Skipped 6 previous similar messages [ 6039.013410] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6039.020221] LustreError: Skipped 1 previous similar message [ 6049.253141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6049.265762] Lustre: Skipped 3 previous similar messages [ 6052.707102] Lustre: server umount lustre-MDT0000 complete [ 6052.718408] Lustre: Skipped 1 previous similar message [ 6057.085850] LustreError: 210571:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197037 with bad export cookie 15060872059675989418 [ 6057.088478] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6057.100953] LustreError: 210571:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6063.571383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6067.656504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6072.031667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6076.365048] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6082.638239] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 16:04:21 (1776197061) [ 6099.761809] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 6107.051771] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 6116.082335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6122.329628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6128.525786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6135.231163] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6142.532114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6142.620620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6142.964511] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6143.003285] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6143.160737] Lustre: lustre-MDT0000: new disk, initializing [ 6143.272853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6143.275639] Lustre: Skipped 3 previous similar messages [ 6143.290879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6148.268963] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 6159.326223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6159.440287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6159.540347] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6159.573512] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6159.576981] Lustre: Skipped 1 previous similar message [ 6159.674653] Lustre: lustre-MDT0001: new disk, initializing [ 6159.844395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6159.847584] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6166.189957] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 6178.296138] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6184.755535] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6192.097314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6192.178393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6192.183967] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6192.422732] Lustre: lustre-OST0000: new disk, initializing [ 6192.433565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6197.281410] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 6197.835942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6197.851274] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6206.402516] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6211.648115] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6211.966296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6217.247674] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6217.465599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6223.842129] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6223.851754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6223.860527] LustreError: Skipped 1 previous similar message [ 6223.861459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6223.861464] Lustre: Skipped 8 previous similar messages [ 6223.900746] Lustre: Skipped 2 previous similar messages [ 6225.843458] Lustre: server umount lustre-OST0000 complete [ 6225.853265] Lustre: Skipped 1 previous similar message [ 6239.201446] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6239.222733] LustreError: Skipped 15 previous similar messages [ 6240.130391] LustreError: 218807:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197220 with bad export cookie 15060872059675991154 [ 6240.149609] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6247.819338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6252.465841] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 6261.699956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6266.981426] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 6275.017864] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6282.618920] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6290.995757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6291.289898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6291.295606] Lustre: Skipped 4 previous similar messages [ 6296.783581] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 6304.696764] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6309.896047] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6310.112647] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6314.645858] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6314.879807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6319.072141] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6319.081946] LustreError: Skipped 2 previous similar messages [ 6329.028820] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6333.632570] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6361.537208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6366.188496] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 6372.131117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6372.149043] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6372.149923] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.119@tcp (at 0@lo) [ 6372.183781] Lustre: Skipped 1 previous similar message [ 6372.198113] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 6373.578362] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6375.324236] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6382.574679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6382.599353] Lustre: Skipped 4 previous similar messages [ 6382.608923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6382.618333] Lustre: Skipped 6 previous similar messages [ 6385.416936] Lustre: server umount lustre-OST0000 complete [ 6385.419233] Lustre: Skipped 3 previous similar messages [ 6397.924579] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6397.946474] LustreError: Skipped 4 previous similar messages [ 6399.414436] LustreError: 226450:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197379 with bad export cookie 15060872059675992008 [ 6399.416912] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6399.432455] LustreError: 226450:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6409.288888] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 16:09:47 (1776197387) [ 6419.730491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6425.078751] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 6433.981201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6440.352478] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 6450.970088] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6459.256872] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6465.938775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6470.043151] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 6479.358653] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6484.214104] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6484.498447] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6489.234838] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6489.453884] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6496.301808] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6496.434034] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6496.451101] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 6496.579559] Lustre: lustre-OST0001: new disk, initializing [ 6496.584117] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 6501.699313] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 6503.512880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6503.531194] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6511.411945] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6516.384764] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 6516.629836] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6521.385803] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 6521.702716] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6524.395281] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6524.405042] LustreError: Skipped 4 previous similar messages [ 6536.717354] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 6536.943411] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6542.229257] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 6542.522269] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6571.973179] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6572.309914] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6572.325580] Lustre: Skipped 5 previous similar messages [ 6576.854524] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 6577.649113] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6577.661842] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6577.672254] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to (at 0@lo) [ 6577.679157] Lustre: Skipped 1 previous similar message [ 6584.852916] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6586.512422] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6626.271111] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6639.077746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6639.086262] Lustre: Skipped 21 previous similar messages [ 6644.191616] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6644.347208] Lustre: server umount lustre-MDT0000 complete [ 6644.353087] Lustre: Skipped 5 previous similar messages [ 6648.085285] LustreError: 229345:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197628 with bad export cookie 15060872059675993198 [ 6648.102630] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6659.036542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6662.651592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6665.812774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6669.173106] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6678.041947] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 16:14:15 (1776197655) [ 6685.773410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6686.224880] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6686.265205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6691.817116] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 6699.850927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6699.973208] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6700.025970] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6704.702880] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 6713.766976] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6722.960875] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6729.443922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6729.647200] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6734.866260] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 6744.621613] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6754.171322] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6754.253418] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6758.684313] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 6766.649169] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6768.367475] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6772.347232] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 6772.559891] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6776.874466] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 6777.026278] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6783.230323] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9673c54ed000.ost_server_uuid 40 [ 6785.371482] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9673c54ed000.ost_server_uuid in IDLE state after 0 sec [ 6786.528652] Lustre: Permanently deactivating lustre-OST0001 [ 6786.541783] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 6790.158931] Lustre: Permanently reactivating lustre-OST0001 [ 6796.770042] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6796.788907] LustreError: Skipped 3 previous similar messages [ 6796.796099] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6796.813326] Lustre: Skipped 11 previous similar messages [ 6797.293674] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6797.316698] LustreError: Skipped 41 previous similar messages [ 6820.734961] LustreError: 238278:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197800 with bad export cookie 15060872059675994472 [ 6820.745170] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6820.749475] LustreError: 238278:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 6826.649661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6830.250847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6833.500931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6836.824720] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6843.853592] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 16:17:02 (1776197822) [ 6844.867467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6851.002876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6851.277791] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6851.313932] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6851.317424] Lustre: Skipped 5 previous similar messages [ 6855.600593] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 6863.289964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6863.370449] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6867.797391] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 6874.788267] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6880.759881] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6887.157534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6887.256491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6887.441626] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6887.471692] Lustre: Permanently deactivating lustre-OST0000 [ 6892.334465] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 6896.188333] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6900.236642] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6906.782614] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6906.910923] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6912.131731] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 6912.671326] Lustre: lustre-OST0001: deleting orphan objects from 0x0:67 to 0x0:97 [ 6919.744466] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6921.591559] Lustre: Permanently reactivating lustre-OST0000 [ 6921.616027] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 6921.620360] Lustre: Skipped 4 previous similar messages [ 6931.502408] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6931.557302] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 6965.216229] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6968.907647] LustreError: 245125:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197948 with bad export cookie 15060872059675996096 [ 6968.912742] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6968.921036] LustreError: 245125:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6981.289479] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 6983.857218] Key type lgssc unregistered [ 6984.195264] LNet: 250094:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6984.219363] LNet: Removed LNI 192.168.203.119@tcp [ 6985.005546] Key type .llcrypt unregistered [ 6985.009632] Key type ._llcrypt unregistered [ 7006.151954] alg: No test for adler32 (adler32-zlib) [ 7006.909897] Key type ._llcrypt registered [ 7006.914382] Key type .llcrypt registered [ 7007.020182] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 7007.999576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7008.475472] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 7008.695106] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 7008.698868] LNet: Accept secure, port 988 [ 7010.367156] Key type lgssc registered [ 7011.346861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7014.720758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7021.574392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7022.906910] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7022.959116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7027.550425] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 7028.195928] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7033.325396] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7036.421906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7036.520513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7036.708551] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7036.746123] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7036.753537] Lustre: Permanently deactivating lustre-MDT0001 [ 7037.095340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7039.160276] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7041.077463] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 7048.835985] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7055.335343] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7061.374697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7061.788970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7066.585308] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 7068.961679] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7074.083097] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7074.102741] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 7074.108814] LustreError: Skipped 1 previous similar message [ 7074.739230] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7081.085906] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7081.277322] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7085.453158] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 7086.587902] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:99 to 0x2c0000400:129 [ 7086.593214] Lustre: lustre-OST0001: deleting orphan objects from 0x0:67 to 0x0:129 [ 7093.551795] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7096.939773] Lustre: Permanently reactivating lustre-MDT0001 [ 7096.963131] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 7096.969942] Lustre: Skipped 3 previous similar messages [ 7102.096963] Lustre: lustre-MDT0001: Client 574995b0-87f0-4cb1-bee7-6ca151e3c46d (at 192.168.203.19@tcp) reconnecting [ 7105.157218] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7105.175117] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7105.190309] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7105.214980] LustreError: 255239:0:(llog_osd.c:2129:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7105.216336] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to (at 0@lo) [ 7105.225266] LustreError: 255239:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7105.225380] LustreError: 255239:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.203.119@tcp: cfg command failed: rc = -108 [ 7105.228240] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7105.228240] [ 7105.241494] LustreError: 251600:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7109.628688] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7111.649887] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7117.464499] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 7117.710938] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7122.491773] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 7122.662654] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7125.187241] Lustre: Permanently deactivating lustre-MDT0001 [ 7125.209824] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 7125.217336] Lustre: Skipped 2 previous similar messages [ 7134.288556] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7142.884756] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7142.892272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7142.899819] Lustre: Skipped 1 previous similar message [ 7142.911748] Lustre: Skipped 3 previous similar messages [ 7148.004046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7148.018393] Lustre: Skipped 3 previous similar messages [ 7151.009327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7156.191182] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7156.193778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7156.206180] Lustre: Skipped 6 previous similar messages [ 7156.328383] Lustre: server umount lustre-MDT0000 complete [ 7160.640293] LustreError: 251588:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198140 with bad export cookie 15968011164521207468 [ 7160.642861] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7160.659836] LustreError: 251588:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7160.974898] Lustre: server umount lustre-MDT0001 complete [ 7166.852581] Lustre: server umount lustre-OST0000 complete [ 7177.798695] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 16:22:36 (1776198156) [ 7190.342960] Lustre: server umount lustre-OST0001 complete [ 7197.962727] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 7208.235444] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 7218.131899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7225.643205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7232.702368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7238.994457] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7245.741022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7245.825688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7246.150543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7246.163584] Lustre: Skipped 2 previous similar messages [ 7246.210291] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7246.400303] Lustre: lustre-MDT0000: new disk, initializing [ 7246.519869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7246.552436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7251.346760] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 7260.925706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7260.988720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7261.067237] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7261.102550] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7261.116465] Lustre: Skipped 1 previous similar message [ 7261.195294] Lustre: lustre-MDT0001: new disk, initializing [ 7261.265796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7261.301651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7261.323151] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7265.703595] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 7274.909312] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7280.511339] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7286.724862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7286.789819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7286.797900] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7286.968170] Lustre: lustre-OST0000: new disk, initializing [ 7286.972889] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7287.052891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7291.011485] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 7295.020442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7295.046409] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7300.119506] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7302.335417] LustreError: 262879:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7310.633789] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7310.701709] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7310.709367] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 7310.906842] Lustre: lustre-OST0001: new disk, initializing [ 7310.910575] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 7312.399185] LustreError: 262879:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 7312.434773] LustreError: 262879:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7315.137419] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 7318.138666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7318.150791] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7322.455861] LustreError: 262879:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 7324.537110] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7326.278728] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7328.738272] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7328.747620] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7328.748414] Lustre: Skipped 2 previous similar messages [ 7328.749677] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7328.776882] LustreError: Skipped 1 previous similar message [ 7333.964654] Lustre: server umount lustre-OST0001 complete [ 7333.985576] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7333.993988] LustreError: Skipped 2 previous similar messages [ 7351.779260] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7351.780579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7351.780589] Lustre: Skipped 1 previous similar message [ 7351.781985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7351.781991] Lustre: Skipped 4 previous similar messages [ 7351.804690] LustreError: Skipped 1 previous similar message [ 7354.343396] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7354.368435] LustreError: Skipped 8 previous similar messages [ 7357.683457] Lustre: server umount lustre-OST0000 complete [ 7363.558385] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7363.565705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7363.570830] Lustre: Skipped 1 previous similar message [ 7369.696670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7369.700037] Lustre: Skipped 7 previous similar messages [ 7375.329453] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7375.402663] Lustre: server umount lustre-MDT0000 complete [ 7378.439698] LustreError: 260114:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198358 with bad export cookie 15968011164521210037 [ 7378.441075] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7378.448254] LustreError: 260114:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7395.353563] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 7397.817863] Key type lgssc unregistered [ 7398.107811] LNet: 265319:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7398.129079] LNet: Removed LNI 192.168.203.119@tcp [ 7398.928407] Key type .llcrypt unregistered [ 7398.931092] Key type ._llcrypt unregistered [ 7429.419231] alg: No test for adler32 (adler32-zlib) [ 7430.170732] Key type ._llcrypt registered [ 7430.174302] Key type .llcrypt registered [ 7430.275674] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 7444.570427] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 7445.795015] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7446.239890] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 7446.417643] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 7446.422633] LNet: Accept secure, port 988 [ 7448.087127] Key type lgssc registered [ 7449.074210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7457.552037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7462.823029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7467.844534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7474.122461] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7480.720220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7480.787508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7482.033171] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7482.062753] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7482.176779] Lustre: lustre-MDT0000: new disk, initializing [ 7482.266922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7482.297080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7486.401949] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 7495.787561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7495.842724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7495.931753] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7495.973388] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7495.978238] Lustre: Skipped 1 previous similar message [ 7496.071928] Lustre: lustre-MDT0001: new disk, initializing [ 7496.174428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7496.215911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7496.231157] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7500.313275] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 7509.404187] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7514.680570] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7521.787679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7521.847891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7521.858489] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7522.117657] Lustre: lustre-OST0000: new disk, initializing [ 7522.124859] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7522.252751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7526.508694] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 7527.972558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7527.984464] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7535.795494] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7540.869820] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7541.279393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7546.441879] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7546.668391] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7553.531921] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7553.563876] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7553.609256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7559.143743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7559.149043] Lustre: Skipped 3 previous similar messages [ 7563.231249] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7563.431421] Lustre: server umount lustre-OST0000 complete [ 7567.840117] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7567.849633] LustreError: Skipped 1 previous similar message [ 7567.853594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7567.876186] Lustre: Skipped 1 previous similar message [ 7567.881266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7569.378268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7572.720657] Lustre: server umount lustre-MDT0000 complete [ 7574.497954] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7574.520764] LustreError: Skipped 1 previous similar message [ 7576.176660] LustreError: 269434:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198556 with bad export cookie 2247924011621960462 [ 7576.185112] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7576.483818] Lustre: server umount lustre-MDT0001 complete [ 7584.096973] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 16:29:22 (1776198562) [ 7589.465049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7589.806625] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7589.841778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7593.430409] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 7594.981396] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7600.101734] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7601.160711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7601.437539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7605.320855] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 7612.155620] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7617.258914] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7622.627498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7622.878471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7626.207448] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 7633.001887] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7660.000242] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7660.003789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7660.005630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7660.005638] Lustre: Skipped 2 previous similar messages [ 7660.062673] Lustre: Skipped 1 previous similar message [ 7665.120892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7665.127812] Lustre: Skipped 1 previous similar message [ 7666.001940] Lustre: server umount lustre-OST0000 complete [ 7670.052780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7679.741103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7679.968368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7683.752965] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 7686.122253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7686.138397] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7686.138710] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 7686.143341] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 7686.180037] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 7686.182580] Lustre: Skipped 1 previous similar message [ 7690.854651] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7701.472694] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7701.482732] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7701.486833] LustreError: Skipped 1 previous similar message [ 7701.495331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7701.518727] Lustre: Skipped 1 previous similar message [ 7701.538324] Lustre: Skipped 2 previous similar messages [ 7707.427741] Lustre: server umount lustre-OST0000 complete [ 7711.713612] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7711.737759] Lustre: Skipped 1 previous similar message [ 7717.276082] Lustre: server umount lustre-MDT0000 complete [ 7720.815591] LustreError: 273976:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198700 with bad export cookie 2247924011621961316 [ 7720.824934] LustreError: 273976:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7720.825969] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7731.850333] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 7734.217338] Key type lgssc unregistered [ 7734.456952] LNet: 279455:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7734.474585] LNet: Removed LNI 192.168.203.119@tcp [ 7735.230513] Key type .llcrypt unregistered [ 7735.235283] Key type ._llcrypt unregistered [ 7757.646627] alg: No test for adler32 (adler32-zlib) [ 7758.404974] Key type ._llcrypt registered [ 7758.411196] Key type .llcrypt registered [ 7758.551062] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 7759.871560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7760.336664] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 7760.538321] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 7760.548117] LNet: Accept secure, port 988 [ 7762.295143] Key type lgssc registered [ 7763.332631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7770.405981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7771.799214] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7771.841339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7776.146519] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 7777.253880] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7782.369665] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7784.264819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7784.639052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7788.413783] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 7795.637397] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7801.253042] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7807.680304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7808.034197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7812.183146] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 7820.162808] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7822.378431] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:97 [ 7844.834333] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7844.847094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7844.867781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7847.908678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7847.909754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7847.927737] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7847.939260] Lustre: Skipped 1 previous similar message [ 7850.223307] Lustre: server umount lustre-OST0000 complete [ 7856.608157] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7856.619385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7856.640309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7859.847714] LustreError: 284540:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000dfd2a3f7) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 7859.911478] Lustre: server umount lustre-MDT0000 complete [ 7863.265594] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7863.283237] LustreError: Skipped 1 previous similar message [ 7863.285975] LustreError: 280898:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198843 with bad export cookie 2720146095785860713 [ 7863.301349] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7863.314745] LustreError: 280898:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7863.641806] Lustre: server umount lustre-MDT0001 complete [ 7872.555691] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 7874.472831] Key type lgssc unregistered [ 7874.691059] LNet: 285344:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7874.702642] LNet: Removed LNI 192.168.203.119@tcp [ 7875.325044] Key type .llcrypt unregistered [ 7875.331018] Key type ._llcrypt unregistered [ 7891.028929] alg: No test for adler32 (adler32-zlib) [ 7891.781449] Key type ._llcrypt registered [ 7891.783891] Key type .llcrypt registered [ 7891.876238] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 7892.852498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7893.271193] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 7893.518347] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 7893.524550] LNet: Accept secure, port 988 [ 7895.183105] Key type lgssc registered [ 7896.115775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7902.177752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7903.465547] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7903.519745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7906.745614] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 7908.833846] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7912.841832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7913.139979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7916.641253] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 7922.634521] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7927.946520] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7934.550649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7934.925583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7938.836774] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 7945.580638] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7953.410742] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 7968.737108] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7968.743485] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7968.752160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7968.756735] Lustre: Skipped 1 previous similar message [ 7973.858795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7973.867861] Lustre: Skipped 2 previous similar messages [ 7974.204987] Lustre: server umount lustre-OST0000 complete [ 7978.979205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7978.988719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7978.991918] Lustre: Skipped 1 previous similar message [ 7978.999410] Lustre: Skipped 1 previous similar message [ 7983.468989] Lustre: server umount lustre-MDT0000 complete [ 7984.096930] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7984.113770] LustreError: Skipped 1 previous similar message [ 7986.615366] LustreError: 286615:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198966 with bad export cookie 10933991889218280328 [ 7986.623252] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7986.624716] LustreError: 286615:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7986.857794] Lustre: server umount lustre-MDT0001 complete [ 7998.832145] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 8001.235442] Key type lgssc unregistered [ 8001.480947] LNet: 290479:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8001.488382] LNet: Removed LNI 192.168.203.119@tcp [ 8002.246861] Key type .llcrypt unregistered [ 8002.249377] Key type ._llcrypt unregistered [ 8015.572353] alg: No test for adler32 (adler32-zlib) [ 8016.333738] Key type ._llcrypt registered [ 8016.337582] Key type .llcrypt registered [ 8016.498282] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 8017.573240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8018.021763] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 8018.235562] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 8018.241992] LNet: Accept secure, port 988 [ 8019.927179] Key type lgssc registered [ 8020.936793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8027.612785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8028.976376] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8029.041526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8032.872041] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 8034.275259] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8039.393817] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8043.129189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8043.575315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8048.542342] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 8056.965483] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8062.641271] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8069.655315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8069.955727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8074.864796] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 8079.168789] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 8083.161481] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8094.688683] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8094.689536] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8094.691323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8094.717173] LustreError: Skipped 1 previous similar message [ 8094.729624] Lustre: Skipped 1 previous similar message [ 8098.089836] Lustre: server umount lustre-OST0000 complete [ 8104.933712] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8104.936144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8104.955984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8104.973939] Lustre: Skipped 1 previous similar message [ 8104.997232] Lustre: Skipped 2 previous similar messages [ 8108.524110] Lustre: server umount lustre-MDT0000 complete [ 8110.051148] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8110.064988] LustreError: Skipped 1 previous similar message [ 8112.582667] LustreError: 292532:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199092 with bad export cookie 7779760716571390841 [ 8112.596363] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8112.938191] Lustre: server umount lustre-MDT0001 complete [ 8124.924993] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 8127.517046] Key type lgssc unregistered [ 8127.801314] LNet: 295404:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8127.813441] LNet: Removed LNI 192.168.203.119@tcp [ 8128.565330] Key type .llcrypt unregistered [ 8128.567582] Key type ._llcrypt unregistered [ 8149.464313] alg: No test for adler32 (adler32-zlib) [ 8150.224668] Key type ._llcrypt registered [ 8150.229536] Key type .llcrypt registered [ 8150.320767] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 8151.251567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8151.583300] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 8151.766472] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 8151.777413] LNet: Accept secure, port 988 [ 8153.400620] Key type lgssc registered [ 8154.249822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8160.898431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8162.382817] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8162.450853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8166.298755] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 8167.911913] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8173.027655] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8174.390842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8174.938034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8179.447859] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 8187.023507] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8193.238519] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8198.914452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8199.196604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8203.093071] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 8210.119152] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8212.538382] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 8236.002520] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8236.013372] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8236.022281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8238.050892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8238.054144] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8238.060949] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8242.148351] Lustre: server umount lustre-OST0000 complete [ 8246.753182] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8246.769235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8246.794430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8246.800584] Lustre: Skipped 1 previous similar message [ 8252.200324] Lustre: server umount lustre-MDT0000 complete [ 8253.426556] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8253.457832] LustreError: Skipped 1 previous similar message [ 8255.928982] LustreError: 296846:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199235 with bad export cookie 1619705658358385638 [ 8255.942667] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8255.944594] LustreError: 296846:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8256.365328] Lustre: server umount lustre-MDT0001 complete [ 8267.460317] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 8269.409453] Key type lgssc unregistered [ 8269.656847] LNet: 301345:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8269.661090] LNet: Removed LNI 192.168.203.119@tcp [ 8270.305965] Key type .llcrypt unregistered [ 8270.308870] Key type ._llcrypt unregistered [ 8284.899181] alg: No test for adler32 (adler32-zlib) [ 8285.654398] Key type ._llcrypt registered [ 8285.658241] Key type .llcrypt registered [ 8285.834323] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 8286.744951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8287.123736] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 8287.357729] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 8287.362694] LNet: Accept secure, port 988 [ 8289.015110] Key type lgssc registered [ 8289.802105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8296.331688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8297.640627] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8297.687507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8300.767438] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 8303.076626] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8306.951161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8307.261808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8310.441879] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 8316.872964] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8322.847467] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8328.190421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8328.429723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8329.449254] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 8332.608153] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 8339.582834] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8349.154624] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8349.157937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8349.167941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8349.192147] Lustre: Skipped 1 previous similar message [ 8354.274853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8354.283302] Lustre: Skipped 2 previous similar messages [ 8355.249848] Lustre: server umount lustre-OST0000 complete [ 8359.394444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8359.397325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8359.414031] Lustre: Skipped 1 previous similar message [ 8359.422209] Lustre: Skipped 1 previous similar message [ 8364.513469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8364.528301] Lustre: Skipped 1 previous similar message [ 8369.633510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8369.645801] Lustre: Skipped 1 previous similar message [ 8373.215201] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8373.429512] Lustre: server umount lustre-MDT0000 complete [ 8374.756278] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8374.771424] LustreError: Skipped 1 previous similar message [ 8377.541954] LustreError: 302596:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199357 with bad export cookie 4397218034002188950 [ 8377.558397] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8377.566812] LustreError: 302596:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8378.311190] Lustre: server umount lustre-MDT0001 complete [ 8392.649237] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 8396.258480] Key type lgssc unregistered [ 8396.677056] LNet: 306452:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8396.684796] LNet: Removed LNI 192.168.203.119@tcp [ 8397.842827] Key type .llcrypt unregistered [ 8397.845430] Key type ._llcrypt unregistered [ 8416.379426] alg: No test for adler32 (adler32-zlib) [ 8417.141781] Key type ._llcrypt registered [ 8417.144438] Key type .llcrypt registered [ 8417.289092] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 8418.383723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8418.944687] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 8419.123575] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 8419.128618] LNet: Accept secure, port 988 [ 8420.759123] Key type lgssc registered [ 8421.566806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8428.760635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8430.132396] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8430.204329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8434.101162] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 8435.686252] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8440.801145] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8441.798975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8442.186956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8446.498957] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 8453.662617] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8459.656575] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8466.351795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8466.702716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8470.858505] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 8478.542894] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8481.080099] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:257 [ 8491.488708] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8491.506952] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8491.515261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8491.526779] Lustre: Skipped 1 previous similar message [ 8496.610706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8496.624163] Lustre: Skipped 2 previous similar messages [ 8497.007980] Lustre: server umount lustre-OST0000 complete [ 8501.730975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8501.733625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8501.741718] Lustre: Skipped 1 previous similar message [ 8501.753979] Lustre: Skipped 1 previous similar message [ 8506.851647] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8506.875385] LustreError: Skipped 1 previous similar message [ 8506.903273] Lustre: server umount lustre-MDT0000 complete [ 8510.415856] LustreError: 307720:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199490 with bad export cookie 5378156756450420238 [ 8510.437425] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8510.728411] Lustre: server umount lustre-MDT0001 complete [ 8522.228090] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 8524.936426] Key type lgssc unregistered [ 8525.184597] LNet: 311408:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8525.198128] LNet: Removed LNI 192.168.203.119@tcp [ 8525.971064] Key type .llcrypt unregistered [ 8525.974290] Key type ._llcrypt unregistered [ 8537.360632] alg: No test for adler32 (adler32-zlib) [ 8538.124197] Key type ._llcrypt registered [ 8538.126176] Key type .llcrypt registered [ 8538.251740] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8555.209807] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 8566.655539] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 8567.673993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8568.015949] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 8568.294959] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 8568.303420] LNet: Accept secure, port 988 [ 8569.983116] Key type lgssc registered [ 8571.085503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8579.180432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8584.758442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8591.017949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8597.578767] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8604.584122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8604.644525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8605.819411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8605.850718] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8605.908437] Lustre: lustre-MDT0000: new disk, initializing [ 8605.999317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8606.020197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8610.385311] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 8621.354432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8621.426421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8621.498525] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8621.567246] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8621.597590] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8621.606063] Lustre: Skipped 1 previous similar message [ 8621.682261] Lustre: lustre-MDT0001: new disk, initializing [ 8621.805605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8621.845953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8621.867600] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8626.379283] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 8635.645773] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8642.098253] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8648.535687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8648.598172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8648.604809] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8648.872733] Lustre: lustre-OST0000: new disk, initializing [ 8648.881877] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8648.986653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8653.262641] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 8657.986630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8658.002407] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8662.230767] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8667.559500] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8667.886194] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8673.961799] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 8674.151241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8678.884333] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8678.885336] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8678.903039] LustreError: Skipped 1 previous similar message [ 8678.927538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8678.932451] Lustre: Skipped 1 previous similar message [ 8682.340203] Lustre: server umount lustre-OST0000 complete [ 8688.608740] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8688.625519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8688.658817] Lustre: Skipped 1 previous similar message [ 8688.672645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8693.715347] Lustre: server umount lustre-MDT0000 complete [ 8694.247242] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8694.274298] LustreError: Skipped 1 previous similar message [ 8697.794129] LustreError: 315154:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199677 with bad export cookie 14604411882595401180 [ 8697.810958] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8697.818231] LustreError: 315154:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8698.258291] Lustre: server umount lustre-MDT0001 complete [ 8707.793787] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 16:48:06 (1776199686) [ 8713.857574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8714.305474] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8714.386610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8719.193958] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 8719.841444] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8724.960507] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8726.817031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8727.203848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8731.745946] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 8739.579992] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8746.200155] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8753.661961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8754.177476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8758.849962] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 8768.287840] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8777.699050] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8777.722635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8777.744085] Lustre: Skipped 1 previous similar message [ 8777.747798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8777.754530] Lustre: Skipped 2 previous similar messages [ 8779.747051] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8779.750780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8779.752248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8779.784787] Lustre: Skipped 1 previous similar message [ 8783.179423] Lustre: server umount lustre-OST0000 complete [ 8788.960295] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8788.972640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8788.982358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8793.307749] Lustre: server umount lustre-MDT0000 complete [ 8795.105831] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8795.127812] LustreError: Skipped 1 previous similar message [ 8797.409689] LustreError: 319699:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199777 with bad export cookie 14604411882595402034 [ 8797.425762] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8809.631266] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 8812.521191] Key type lgssc unregistered [ 8812.858845] LNet: 323377:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8812.871321] LNet: Removed LNI 192.168.203.119@tcp [ 8813.763410] Key type .llcrypt unregistered [ 8813.766631] Key type ._llcrypt unregistered [ 8840.471410] alg: No test for adler32 (adler32-zlib) [ 8841.230548] Key type ._llcrypt registered [ 8841.232509] Key type .llcrypt registered [ 8841.392556] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 8853.039551] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 8854.037966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8854.643184] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 8854.954111] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 8854.958300] LNet: Accept secure, port 988 [ 8856.591860] Key type lgssc registered [ 8857.867369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8867.265842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8874.567938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8880.431927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8887.329174] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8892.697993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8897.222829] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8903.467349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8903.559886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8904.787726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8904.840059] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8904.961623] Lustre: lustre-MDT0000: new disk, initializing [ 8905.077433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8905.116866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8909.366847] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 8920.280923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8920.408763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8920.597326] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8920.717773] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8920.730441] Lustre: Skipped 1 previous similar message [ 8920.871638] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8920.877676] Lustre: lustre-MDT0001: new disk, initializing [ 8920.990100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8921.038375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8921.054284] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8925.858991] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 8937.713922] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8943.105156] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8949.854712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8949.969542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8949.978742] Lustre: lustre-OST2710-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8950.171876] Lustre: lustre-OST2710: new disk, initializing [ 8950.176406] Lustre: srv-lustre-OST2710: No data found on store. Initialize space: rc = -61 [ 8950.240898] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8954.071968] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 8958.581441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8958.612344] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8962.857186] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8969.323651] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8969.388541] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8969.397586] Lustre: lustre-OST03e8-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 8969.574332] Lustre: lustre-OST03e8: new disk, initializing [ 8969.581417] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space: rc = -61 [ 8969.669193] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8973.444808] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 8976.004081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8976.024049] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8981.329555] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8988.175676] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 40 [ 8988.460194] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8992.687267] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 40 [ 8992.923870] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8997.817231] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 40 [ 8998.066138] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9002.525801] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 40 [ 9002.725803] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9007.078571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9007.080125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9007.098683] Lustre: Skipped 3 previous similar messages [ 9012.202639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9012.208100] Lustre: Skipped 6 previous similar messages [ 9012.622934] Lustre: server umount lustre-MDT0000 complete [ 9016.174184] LustreError: 327336:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199996 with bad export cookie 11064773245328830784 [ 9016.180958] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9016.188780] LustreError: 327336:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9016.574896] Lustre: server umount lustre-MDT0001 complete [ 9026.527125] Lustre: 332831:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776200000/real 1776200000] req@000000005ce9d876 x1862480517168512/t0(0) o39->lustre-MDT0001-lwp-OST2710@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776200006 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9026.605390] Lustre: server umount lustre-OST2710 complete [ 9036.769399] Lustre: 333032:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776200010/real 1776200010] req@000000005ce9d876 x1862480517168896/t0(0) o39->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776200016 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9036.965879] Lustre: server umount lustre-OST03e8 complete [ 9056.049861] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 9064.723569] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 9073.765812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9080.705398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9086.615726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9092.968884] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9101.077902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9101.152276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9101.516498] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9101.552767] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9101.753172] Lustre: lustre-MDT0000: new disk, initializing [ 9101.915453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9101.948385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9106.958442] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 9118.574656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9118.682589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9118.835821] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9118.887863] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9118.892283] Lustre: Skipped 1 previous similar message [ 9118.974789] Lustre: lustre-MDT0001: new disk, initializing [ 9119.239252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9119.288621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9119.302399] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9124.471797] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 9137.288404] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9144.054892] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9151.594123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9151.652848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9151.658129] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 9151.869632] Lustre: lustre-OST0000: new disk, initializing [ 9151.873903] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9151.957556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9156.962944] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 9162.385251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9162.409199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9170.378703] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9176.943789] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 9177.240927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9181.654679] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 9181.962524] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9188.331282] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9188.336960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9188.338277] LustreError: Skipped 1 previous similar message [ 9188.340510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9188.356363] Lustre: Skipped 1 previous similar message [ 9188.372208] Lustre: Skipped 1 previous similar message [ 9189.865049] Lustre: server umount lustre-OST0000 complete [ 9196.003118] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9196.013728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9196.034861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9198.567073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9204.707820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9204.717136] Lustre: Skipped 5 previous similar messages [ 9208.287210] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 9208.381067] Lustre: server umount lustre-MDT0000 complete [ 9209.827580] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9211.506257] LustreError: 335940:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200191 with bad export cookie 11064773245328832401 [ 9211.509020] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9211.514634] LustreError: 335940:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9222.149872] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 16:56:40 (1776200200) [ 9223.948427] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9225.754440] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 16:56:44 (1776200204) [ 9230.441708] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 9240.294655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9244.680635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9248.514359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9252.280597] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9253.223361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9260.098794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9260.548559] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9260.580290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9260.821304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9265.268090] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 9274.360980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9274.434843] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9274.472439] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9279.329412] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 9287.087275] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9293.083859] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9300.049120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9300.131986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9300.310287] Lustre: Denying initial registration attempt from nid 192.168.203.119@tcp, specified as failover [ 9300.316148] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9300.323376] LustreError: 343974:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -99 [ 9300.332101] LustreError: 343974:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 9300.342200] LustreError: 343974:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 9300.464994] Lustre: server umount lustre-OST0000 complete [ 9300.467833] Lustre: Skipped 1 previous similar message [ 9300.472707] LustreError: 343974:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9305.568606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9305.570208] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9305.585503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9305.605869] Lustre: Skipped 1 previous similar message [ 9315.235632] LustreError: 342013:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200295 with bad export cookie 11064773245328833255 [ 9315.261733] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9326.983519] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 9329.376799] Key type lgssc unregistered [ 9329.654268] LNet: 345050:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9329.663632] LNet: Removed LNI 192.168.203.119@tcp [ 9330.457920] Key type .llcrypt unregistered [ 9330.462645] Key type ._llcrypt unregistered [ 9342.609598] alg: No test for adler32 (adler32-zlib) [ 9343.371187] Key type ._llcrypt registered [ 9343.374505] Key type .llcrypt registered [ 9343.489069] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 9344.391629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9344.875966] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 9345.102633] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 9345.111633] LNet: Accept secure, port 988 [ 9346.768226] Key type lgssc registered [ 9348.075985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9355.771581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9360.709212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9365.115719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9369.109068] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9369.985646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9383.929439] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 9393.381409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9394.722793] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9394.777707] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9395.003226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9400.677595] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 9414.477689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9414.771476] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9414.853127] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9415.505471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9422.689508] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 9434.102974] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9439.943704] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9447.241611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9447.313718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9447.568552] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9447.804101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9452.393712] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 9460.236353] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9466.856885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9466.868687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9466.886030] Lustre: Skipped 1 previous similar message [ 9466.894812] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9468.438501] Lustre: server umount lustre-OST0000 complete [ 9471.974366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9471.980267] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9471.981850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9471.981856] Lustre: Skipped 1 previous similar message [ 9471.987509] Lustre: Skipped 1 previous similar message [ 9477.089282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9477.100918] Lustre: Skipped 2 previous similar messages [ 9482.210336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9482.231757] Lustre: Skipped 1 previous similar message [ 9486.303141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 9487.329067] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9487.347585] LustreError: Skipped 1 previous similar message [ 9488.430152] Lustre: server umount lustre-MDT0000 complete [ 9492.461461] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9492.477638] LustreError: Skipped 1 previous similar message [ 9492.623575] LustreError: 347920:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200472 with bad export cookie 8086187740401455354 [ 9492.629500] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9492.631724] LustreError: 347920:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9493.087990] Lustre: server umount lustre-MDT0001 complete [ 9504.053882] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 9507.107829] Key type lgssc unregistered [ 9507.438564] LNet: 351640:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9507.448353] LNet: Removed LNI 192.168.203.119@tcp [ 9508.331794] Key type .llcrypt unregistered [ 9508.335377] Key type ._llcrypt unregistered [ 9537.367362] alg: No test for adler32 (adler32-zlib) [ 9538.129710] Key type ._llcrypt registered [ 9538.131422] Key type .llcrypt registered [ 9538.232242] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 9539.241525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9539.819174] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 9540.077507] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 9540.084781] LNet: Accept secure, port 988 [ 9541.799114] Key type lgssc registered [ 9542.760531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9551.250237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9552.671945] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9552.729201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9557.088485] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 9557.985126] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9563.105698] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9565.263979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9565.578720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9570.062249] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 9577.357794] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9583.015799] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9590.435618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9590.792701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9595.751206] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 9604.191756] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9613.289788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9613.286047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9613.298860] Lustre: Skipped 1 previous similar message [ 9613.314744] Lustre: Skipped 2 previous similar messages [ 9616.501542] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.19@tcp (stopping) [ 9616.509359] Lustre: Skipped 1 previous similar message [ 9618.405287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9618.413384] Lustre: Skipped 2 previous similar messages [ 9619.265948] Lustre: server umount lustre-MDT0000 complete [ 9621.592379] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9623.068560] LustreError: 353105:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200602 with bad export cookie 134523379926237291 [ 9623.074438] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9623.091489] LustreError: 353105:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9623.521478] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9623.537148] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9626.709750] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9626.729345] LustreError: Skipped 1 previous similar message [ 9628.641920] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9628.650757] Lustre: Skipped 1 previous similar message [ 9629.630407] Lustre: server umount lustre-MDT0001 complete [ 9636.111370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9645.079423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9656.304169] Lustre: Evicted from MGS (at 192.168.203.119@tcp) after server handle changed from 0x1ddec4d227d006b to 0x1ddec4d227d1c4f [ 9656.320273] Lustre: MGC192.168.203.119@tcp: Connection restored to 192.168.203.119@tcp (at 0@lo) [ 9656.585192] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9656.602469] LustreError: Skipped 2 previous similar messages [ 9656.676906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9656.721076] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:193 [ 9660.910933] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 9670.211567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9670.525736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9675.772566] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 9676.694714] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 9679.909368] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 9685.142608] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9686.986231] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9692.799861] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9694.543285] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9700.327140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9700.340438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9700.341980] Lustre: Skipped 1 previous similar message [ 9700.359020] Lustre: Skipped 2 previous similar messages [ 9705.886539] Lustre: server umount lustre-OST0000 complete [ 9710.570605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9710.587398] Lustre: Skipped 1 previous similar message [ 9715.864318] Lustre: server umount lustre-MDT0000 complete [ 9719.676647] LustreError: 353105:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200699 with bad export cookie 134523379926244431 [ 9719.678246] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9719.686525] LustreError: 353105:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9720.123734] Lustre: server umount lustre-MDT0001 complete [ 9731.866387] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 9734.887646] Key type lgssc unregistered [ 9735.187866] LNet: 359620:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9735.201870] LNet: Removed LNI 192.168.203.119@tcp [ 9736.113934] Key type .llcrypt unregistered [ 9736.118832] Key type ._llcrypt unregistered [ 9760.077886] alg: No test for adler32 (adler32-zlib) [ 9760.835564] Key type ._llcrypt registered [ 9760.841926] Key type .llcrypt registered [ 9760.979633] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 9762.062101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9762.436903] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 9762.873449] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 9762.882583] LNet: Accept secure, port 988 [ 9764.671331] Key type lgssc registered [ 9765.956665] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9774.110790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9775.556089] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9775.615895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9780.708813] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9781.122220] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 9785.828620] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9790.949725] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9791.223033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9791.756933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9796.605057] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 9805.508783] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9810.795514] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9817.568798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9817.581117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9817.589766] Lustre: Skipped 1 previous similar message [ 9821.668763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9821.684799] Lustre: Skipped 1 previous similar message [ 9826.785870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9826.793099] Lustre: Skipped 1 previous similar message [ 9828.319137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 9828.396040] Lustre: server umount lustre-MDT0000 complete [ 9831.912654] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9831.923936] LustreError: Skipped 1 previous similar message [ 9832.250964] LustreError: 361084:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200812 with bad export cookie 2525488834559304291 [ 9832.259600] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9832.447570] Lustre: server umount lustre-MDT0001 complete [ 9850.673145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9851.150455] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9851.260386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9851.447218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9859.243795] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 9870.442550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9870.583743] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9870.657899] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9870.985497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9876.419420] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 9885.363923] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9891.991841] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9900.841717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9901.114502] LustreError: 13b-9: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9901.145243] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9901.157550] LustreError: 365806:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 9901.169821] LustreError: 365806:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 9901.178336] LustreError: 365806:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 9901.309645] Lustre: server umount lustre-OST0000 complete [ 9901.313218] LustreError: 365806:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9908.398130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9908.771821] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9908.964927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9913.306763] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 9919.186798] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:225 [ 9920.840339] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9929.105848] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9929.195977] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9929.208526] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 9929.299765] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9929.413768] Lustre: lustre-OST0001: new disk, initializing [ 9929.423667] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 9929.539058] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9934.146550] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [ 9935.941486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9935.949904] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9940.725903] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9946.594141] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9946.599218] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9946.611395] LustreError: Skipped 1 previous similar message [ 9946.618502] Lustre: Skipped 1 previous similar message [ 9946.619686] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9946.631720] Lustre: Skipped 1 previous similar message [ 9951.738831] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9951.742835] Lustre: Skipped 1 previous similar message [ 9957.856338] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 9957.973064] Lustre: server umount lustre-OST0001 complete [ 9961.962054] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9961.986163] LustreError: Skipped 1 previous similar message [ 9965.026181] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9965.045000] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9965.077131] Lustre: Skipped 1 previous similar message [ 9965.080624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9965.097206] Lustre: Skipped 2 previous similar messages [ 9977.311684] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 9977.471403] Lustre: server umount lustre-OST0000 complete [ 9982.433938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9982.461290] Lustre: Skipped 1 previous similar message [ 9982.475418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9982.491626] Lustre: Skipped 8 previous similar messages [ 9987.486193] Lustre: server umount lustre-MDT0000 complete [ 9987.577568] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9987.604818] LustreError: Skipped 6 previous similar messages [ 9991.637504] LustreError: 363800:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200971 with bad export cookie 2525488834559304844 [ 9991.641632] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9991.642859] LustreError: 363800:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 8 previous similar messages [10005.300258] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [10008.751524] Key type lgssc unregistered [10009.065636] LNet: 369177:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10009.103598] LNet: Removed LNI 192.168.203.119@tcp [10010.219744] Key type .llcrypt unregistered [10010.232348] Key type ._llcrypt unregistered [10043.454728] alg: No test for adler32 (adler32-zlib) [10044.209677] Key type ._llcrypt registered [10044.211765] Key type .llcrypt registered [10044.281655] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [10057.929081] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [10058.738995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10058.996170] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [10059.168921] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [10059.173938] LNet: Accept secure, port 988 [10060.807135] Key type lgssc registered [10061.548917] Lustre: Echo OBD driver; http://www.lustre.org/ [10068.209461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10074.189372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10079.275803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10084.217616] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10089.998140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10090.060289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10091.308671] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10091.337730] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [10091.441874] Lustre: lustre-MDT0000: new disk, initializing [10091.625871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10091.650642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10097.116046] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [10107.239037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10107.319450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10107.383727] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10107.411788] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [10107.416362] Lustre: Skipped 1 previous similar message [10107.478453] Lustre: lustre-MDT0001: new disk, initializing [10107.588421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10107.623121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10107.640029] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10111.460642] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [10120.687032] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10126.398736] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10133.308735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10133.383103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10133.394867] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [10133.619316] Lustre: lustre-OST0000: new disk, initializing [10133.623095] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [10133.749700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10137.493054] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [10139.178765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10139.188286] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10145.901743] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10149.878074] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [10150.069668] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10154.565369] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [10154.789870] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10159.591230] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10159.593358] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10159.594870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10159.621923] Lustre: Skipped 1 previous similar message [10162.683964] Lustre: server umount lustre-OST0000 complete [10169.312547] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10169.318472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10169.331536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10169.336341] Lustre: Skipped 1 previous similar message [10172.548844] Lustre: server umount lustre-MDT0000 complete [10174.945870] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10174.970357] LustreError: Skipped 1 previous similar message [10175.975519] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10175.995667] LustreError: Skipped 1 previous similar message [10176.668391] LustreError: 373280:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201156 with bad export cookie 16663664857078882719 [10176.677497] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10177.028456] Lustre: server umount lustre-MDT0001 complete [10186.804668] Lustre: DEBUG MARKER: == conf-sanity test 60: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 17:12:45 (1776201165) [10192.380125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10197.662239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10220.910853] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [10230.433507] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [10240.802818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10247.156956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10253.252293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10259.867297] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10265.910042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10265.971327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10266.232734] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10266.273589] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [10266.344134] Lustre: lustre-MDT0000: new disk, initializing [10266.412392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10266.425185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10270.830232] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [10280.715728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10280.771566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10280.833572] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10280.870109] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [10280.874535] Lustre: Skipped 1 previous similar message [10280.939287] Lustre: lustre-MDT0001: new disk, initializing [10281.019045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10281.066269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10281.069809] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10285.117370] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [10294.115950] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10299.248092] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10304.756101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10304.822252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10304.832530] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [10305.009757] Lustre: lustre-OST0000: new disk, initializing [10305.013775] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [10305.077472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10308.950603] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [10314.292206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10314.304719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10317.142660] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10321.851844] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [10322.089332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10325.584658] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [10325.736896] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10330.083239] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10330.088299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10330.089443] LustreError: Skipped 1 previous similar message [10330.089900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10330.089904] Lustre: Skipped 1 previous similar message [10330.114303] Lustre: Skipped 2 previous similar messages [10333.445816] Lustre: server umount lustre-OST0000 complete [10340.320746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10340.331750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10340.346209] Lustre: Skipped 2 previous similar messages [10344.563808] Lustre: server umount lustre-MDT0000 complete [10345.440235] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10345.466173] LustreError: Skipped 1 previous similar message [10349.401882] LustreError: 381047:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201329 with bad export cookie 16663664857078883573 [10349.405095] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10349.427846] LustreError: 381047:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [10358.161384] Lustre: DEBUG MARKER: == conf-sanity test 61: large xattr ====================== 17:15:37 (1776201337) [10365.941957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10366.438666] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10366.449556] LustreError: Skipped 2 previous similar messages [10366.519567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10371.155801] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [10371.553353] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10379.239222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10383.494324] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [10390.114684] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10395.863769] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10402.091742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10402.358565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10402.365042] Lustre: Skipped 1 previous similar message [10406.692215] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [10413.251308] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10424.676577] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61.conf-sanity [10426.443311] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61.conf-sanity [10427.902483] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61.conf-sanity [10429.860929] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61.conf-sanity after remounting MDS [10431.645236] Lustre: Failing over lustre-MDT0000 [10431.856118] Lustre: server umount lustre-MDT0000 complete [10431.862767] Lustre: Skipped 1 previous similar message [10431.970917] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10431.980731] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10431.986286] Lustre: Skipped 2 previous similar messages [10432.034208] LustreError: Skipped 3 previous similar messages [10444.127156] Lustre: 371509:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201417/real 1776201417] req@0000000013c81a27 x1862481780723392/t0(0) o400->MGC192.168.203.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776201424 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [10444.157310] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10449.382696] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10449.401800] LustreError: Skipped 16 previous similar messages [10451.248605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10461.669145] Lustre: Evicted from MGS (at 192.168.203.119@tcp) after server handle changed from 0xe7413ae64e58e84b to 0xe7413ae64e58ec3b [10461.685309] Lustre: MGC192.168.203.119@tcp: Connection restored to 192.168.203.119@tcp (at 0@lo) [10461.975085] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10463.829195] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10466.727172] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [10467.311473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [10467.325679] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10467.387395] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [10476.561551] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10478.445261] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10481.881967] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61.conf-sanity [10483.770960] Lustre: Failing over lustre-MDT0000 [10483.979507] Lustre: server umount lustre-MDT0000 complete [10484.346973] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10484.361157] LustreError: Skipped 15 previous similar messages [10487.776082] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10487.778700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10487.808106] Lustre: Skipped 3 previous similar messages [10494.944065] Lustre: 371509:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201467/real 1776201467] req@000000009a26132b x1862481780734080/t0(0) o400->MGC192.168.203.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776201474 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [10494.980486] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10503.136400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10512.364799] Lustre: Evicted from MGS (at 192.168.203.119@tcp) after server handle changed from 0xe7413ae64e58ec3b to 0xe7413ae64e58ef3d [10512.386479] Lustre: MGC192.168.203.119@tcp: Connection restored to (at 0@lo) [10512.389334] Lustre: Skipped 2 previous similar messages [10512.642985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10512.648419] Lustre: Skipped 1 previous similar message [10512.679460] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10512.980545] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10517.252302] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [10517.995183] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [10518.017696] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [10518.074828] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [10526.668858] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10528.660500] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10538.465527] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10538.466738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10538.467782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10538.505509] Lustre: Skipped 2 previous similar messages [10540.603926] Lustre: server umount lustre-MDT0000 complete [10543.886822] LustreError: 385686:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201523 with bad export cookie 16663664857078886205 [10543.893899] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10543.902907] LustreError: 385686:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [10554.337273] Lustre: 391519:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201528/real 1776201528] req@00000000afa171ea x1862481780746560/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776201534 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [10571.935794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10572.265288] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10572.275206] LustreError: Skipped 33 previous similar messages [10576.878101] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [10585.464577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10589.922819] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [10598.081812] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10604.131104] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10610.790358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10615.094527] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [10622.404337] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [10622.619934] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10627.555022] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10627.577745] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10627.578581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10627.602785] Lustre: Skipped 3 previous similar messages [10629.541030] Lustre: server umount lustre-OST0000 complete [10629.547691] Lustre: Skipped 2 previous similar messages [10643.789022] LustreError: 392326:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201623 with bad export cookie 16663664857078886940 [10643.797596] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10643.802313] LustreError: 392326:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [10655.022552] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [10657.248756] Key type lgssc unregistered [10657.533104] LNet: 395999:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10657.544483] LNet: Removed LNI 192.168.203.119@tcp [10658.305441] Key type .llcrypt unregistered [10658.307544] Key type ._llcrypt unregistered [10680.205539] alg: No test for adler32 (adler32-zlib) [10680.960589] Key type ._llcrypt registered [10680.962763] Key type .llcrypt registered [10681.079396] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [10681.863192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10682.393367] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [10682.584874] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [10682.587978] LNet: Accept secure, port 988 [10684.239270] Key type lgssc registered [10685.105928] Lustre: Echo OBD driver; http://www.lustre.org/ [10691.716904] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10691.724742] LustreError: 397486:0:(osd_handler.c:8215:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10691.737618] LustreError: 397486:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000-osd failed (-22) [10691.743084] LustreError: 397486:0:(obd_mount.c:200:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10691.751950] LustreError: 397486:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10691.763863] LustreError: 397486:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [10698.113902] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10698.120207] LustreError: 397884:0:(osd_handler.c:8215:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10698.128491] LustreError: 397884:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-22) [10698.134958] LustreError: 397884:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10698.143302] LustreError: 397884:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10698.153288] LustreError: 397884:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [10715.027346] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [10717.694476] Key type lgssc unregistered [10717.925469] LNet: 398841:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10717.941131] LNet: Removed LNI 192.168.203.119@tcp [10718.786335] Key type .llcrypt unregistered [10718.792090] Key type ._llcrypt unregistered [10738.701885] alg: No test for adler32 (adler32-zlib) [10739.453561] Key type ._llcrypt registered [10739.456802] Key type .llcrypt registered [10739.569478] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [10753.836055] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [10754.883976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10755.740732] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [10756.040278] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [10756.048712] LNet: Accept secure, port 988 [10757.791313] Key type lgssc registered [10758.922628] Lustre: Echo OBD driver; http://www.lustre.org/ [10768.430686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10775.646407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10782.126243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10788.953847] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10795.835117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10795.914169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10797.173704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10797.198142] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [10797.263811] Lustre: lustre-MDT0000: new disk, initializing [10797.357849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10797.389852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10802.124714] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [10812.560444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10812.632653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10812.737453] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10812.791227] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [10812.801514] Lustre: Skipped 1 previous similar message [10812.875061] Lustre: lustre-MDT0001: new disk, initializing [10812.967241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10813.013651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10813.024363] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10817.801182] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [10827.728369] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10833.760394] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10840.417938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10840.476362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10840.481974] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [10840.741819] Lustre: lustre-OST0000: new disk, initializing [10840.747262] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [10840.848094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10845.518300] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [10848.317714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10848.326051] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10854.405063] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10858.885881] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [10859.082613] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10863.449372] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [10863.677872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10869.219710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10869.231495] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10869.233706] Lustre: Skipped 1 previous similar message [10869.234334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10869.253463] LustreError: Skipped 1 previous similar message [10871.433331] Lustre: server umount lustre-OST0000 complete [10879.464655] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10879.467623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10879.471934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10879.471943] Lustre: Skipped 1 previous similar message [10879.495332] Lustre: Skipped 1 previous similar message [10881.272315] Lustre: server umount lustre-MDT0000 complete [10884.578677] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10884.592761] LustreError: Skipped 1 previous similar message [10885.145752] LustreError: 402130:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201865 with bad export cookie 6487324072123764950 [10885.152554] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10885.158780] LustreError: 402130:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [10885.428336] Lustre: server umount lustre-MDT0001 complete [10893.631234] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10895.627572] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 17:24:34 (1776201874) [10902.309160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10902.870292] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10902.932341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10907.353455] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [10908.136673] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10913.248802] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10915.429702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10915.744866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10919.608684] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [10926.162593] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10931.091342] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10936.643858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10936.893993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10941.543025] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [10949.586672] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10955.680378] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10955.759660] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10955.768483] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [10955.959349] Lustre: lustre-OST0001: new disk, initializing [10955.965070] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [10956.039374] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10961.047086] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [10963.001870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10963.022797] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10971.112273] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10977.909241] Lustre: lustre-OST0001: Not available for connect from 192.168.203.19@tcp (stopping) [10977.922941] Lustre: Skipped 1 previous similar message [10978.801480] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10978.810147] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10980.874481] Lustre: server umount lustre-OST0001 complete [10983.013089] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10994.166606] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10994.184997] LustreError: Skipped 5 previous similar messages [10999.266187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10999.297975] Lustre: Skipped 1 previous similar message [10999.310778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10999.318131] Lustre: Skipped 3 previous similar messages [11004.385257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11004.397977] Lustre: Skipped 1 previous similar message [11004.836238] Lustre: server umount lustre-OST0000 complete [11009.514820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11009.528019] Lustre: Skipped 1 previous similar message [11014.629246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11014.647638] Lustre: Skipped 3 previous similar messages [11023.328551] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [11023.480487] Lustre: server umount lustre-MDT0000 complete [11024.864671] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11024.875309] LustreError: Skipped 5 previous similar messages [11026.854050] LustreError: 408692:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202006 with bad export cookie 6487324072123765804 [11026.860570] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11026.868622] LustreError: 408692:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [11041.247128] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [11041.622503] Lustre: server umount lustre-MDT0001 complete [11052.473617] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [11055.525771] Key type lgssc unregistered [11055.841937] LNet: 411648:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11055.860363] LNet: Removed LNI 192.168.203.119@tcp [11056.739473] Key type .llcrypt unregistered [11056.744056] Key type ._llcrypt unregistered [11090.062361] alg: No test for adler32 (adler32-zlib) [11090.825029] Key type ._llcrypt registered [11090.828027] Key type .llcrypt registered [11091.010568] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [11109.787535] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [11110.738760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11111.363741] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [11111.570308] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [11111.576782] LNet: Accept secure, port 988 [11113.271125] Key type lgssc registered [11114.436734] Lustre: Echo OBD driver; http://www.lustre.org/ [11125.548564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11132.796534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11140.149200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11146.798360] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11155.052263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11155.131903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11156.394332] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [11156.411506] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [11156.526474] Lustre: lustre-MDT0000: new disk, initializing [11156.660582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11156.690613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11161.685164] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [11173.612646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11173.690199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11173.861982] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [11173.922691] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [11173.932877] Lustre: Skipped 1 previous similar message [11174.071920] Lustre: lustre-MDT0001: new disk, initializing [11174.148289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11174.185917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11174.204406] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11179.171784] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [11189.784826] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11195.728866] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11202.008928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11202.075419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11202.081411] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [11202.283941] Lustre: lustre-OST0000: new disk, initializing [11202.291308] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [11202.382170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11206.488692] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [11210.890413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11210.894032] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11215.246988] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11220.280076] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [11220.498295] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11224.073143] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [11224.315462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11226.597322] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11226.597903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11226.606234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11226.623800] Lustre: Skipped 1 previous similar message [11226.642354] Lustre: Skipped 1 previous similar message [11231.713444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11231.736480] Lustre: Skipped 1 previous similar message [11232.225289] Lustre: server umount lustre-OST0000 complete [11240.929674] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11240.944301] LustreError: Skipped 1 previous similar message [11240.954385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11240.977540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11243.321915] Lustre: server umount lustre-MDT0000 complete [11247.072417] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11247.085418] LustreError: Skipped 1 previous similar message [11248.363814] LustreError: 415762:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202228 with bad export cookie 15044025032272724907 [11248.364735] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11248.375686] LustreError: 415762:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [11248.638981] Lustre: server umount lustre-MDT0001 complete [11258.048569] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 17:30:36 (1776202236) [11267.806452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11275.673769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11276.077720] Lustre: lustre-MDT0000: new disk, initializing [11276.109365] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11276.202181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11280.452989] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [11281.380052] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11286.508601] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11288.353758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11288.702277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11292.480688] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [11300.306298] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11305.974446] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11309.026642] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11309.047134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11309.055220] Lustre: Skipped 1 previous similar message [11309.061476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11309.067629] Lustre: Skipped 2 previous similar messages [11312.102102] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11317.217196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11317.228156] Lustre: Skipped 3 previous similar messages [11322.337859] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [11322.400207] Lustre: server umount lustre-MDT0000 complete [11323.364307] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11323.381995] LustreError: Skipped 1 previous similar message [11326.012864] LustreError: 420895:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202305 with bad export cookie 15044025032272725761 [11326.014816] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11326.031536] LustreError: 420895:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [11335.650446] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 17:31:54 (1776202314) [11341.746469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11342.198571] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11342.268224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11346.936424] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [11354.557864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11360.377634] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [11368.374796] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11375.696472] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11382.191894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11382.463871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11382.471249] Lustre: Skipped 1 previous similar message [11386.888336] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [11394.087575] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11400.802581] Lustre: Permanently deactivating lustre-OST0000 [11400.822968] Lustre: Setting parameter lustre-OST0000-osc.osc.active in log lustre-client [11410.466423] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11411.379961] LustreError: 426609:0:(mgs_llog.c:1551:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002020 [11411.392082] LustreError: 426609:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [11411.396287] LustreError: 426609:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [11414.467318] LustreError: 426658:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [11414.472175] LustreError: 426658:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [11422.671066] Lustre: server umount lustre-OST0000 complete [11422.673756] Lustre: Skipped 1 previous similar message [11425.688745] LustreError: 426908:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [11425.698573] LustreError: 426908:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [11430.881516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11430.884727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11430.891674] Lustre: Skipped 1 previous similar message [11430.894611] Lustre: Skipped 2 previous similar messages [11433.951278] Lustre: 427005:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202407/real 1776202407] req@0000000049c94e3e x1862482883820928/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776202413 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [11434.214805] Lustre: server umount lustre-MDT0000 complete [11436.004098] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11436.020181] LustreError: Skipped 3 previous similar messages [11437.744644] LustreError: 423496:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202417 with bad export cookie 15044025032272726314 [11437.746313] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11437.751103] LustreError: 423496:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [11443.167207] Lustre: 427208:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202417/real 1776202417] req@00000000a2178e1a x1862482883821952/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776202423 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [11451.776420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11455.634223] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [11457.903740] LustreError: 428118:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -6 [11463.278745] Lustre: 428463:0:(mgs_llog.c:1311:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.119@tcp [11471.327150] Lustre: 428563:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202445/real 1776202445] req@000000006ed28286 x1862482883822464/t0(0) o251->MGC192.168.203.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776202451 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [11471.364887] Lustre: server umount MGS complete [11471.366772] Lustre: Skipped 1 previous similar message [11482.559502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11483.265299] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11483.319684] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11483.324504] Lustre: Skipped 1 previous similar message [11483.403907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11487.831562] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [11495.730572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11495.986958] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11500.385833] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [11508.187175] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11513.840986] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11515.689551] Lustre: Permanently reactivating lustre-OST0000 [11515.708813] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [11515.715896] Lustre: Skipped 2 previous similar messages [11542.496087] LustreError: 431079:0:(osp_dev.c:684:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11549.470302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11549.836629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11549.847523] Lustre: Skipped 1 previous similar message [11554.106440] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [11560.811214] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11571.686507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11571.689480] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11571.706818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11571.721751] Lustre: Skipped 1 previous similar message [11577.108306] Lustre: server umount lustre-OST0000 complete [11594.207737] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [11597.282148] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11597.288561] LustreError: Skipped 3 previous similar messages [11598.000838] LustreError: 429171:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202577 with bad export cookie 15044025032272727273 [11598.010038] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11598.016483] LustreError: 429171:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [11609.391732] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [11611.943769] Key type lgssc unregistered [11612.202632] LNet: 433724:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11612.214955] LNet: Removed LNI 192.168.203.119@tcp [11613.015540] Key type .llcrypt unregistered [11613.017762] Key type ._llcrypt unregistered [11631.092138] alg: No test for adler32 (adler32-zlib) [11631.843432] Key type ._llcrypt registered [11631.845624] Key type .llcrypt registered [11631.930954] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [11643.728558] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [11644.490498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11645.144582] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [11645.378897] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [11645.383782] LNet: Accept secure, port 988 [11647.063115] Key type lgssc registered [11647.845271] Lustre: Echo OBD driver; http://www.lustre.org/ [11655.351323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11661.308503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11666.589323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11672.666633] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11678.824862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11678.902564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11680.156744] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [11680.208208] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [11680.329691] Lustre: lustre-MDT0000: new disk, initializing [11680.447693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11680.476646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11685.424312] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [11696.154923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11696.239831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11696.377810] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [11696.412688] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [11696.420613] Lustre: Skipped 1 previous similar message [11696.506337] Lustre: lustre-MDT0001: new disk, initializing [11696.636653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11696.681733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11696.696972] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11701.193838] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [11710.253903] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11715.761768] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11722.570978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11722.648399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11722.654972] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [11722.920903] Lustre: lustre-OST0000: new disk, initializing [11722.924140] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [11722.993450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11726.950469] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [11732.046510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11732.063940] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11735.839646] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11740.063835] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [11740.246582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11744.625280] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [11744.850563] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11747.817331] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11747.818631] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11747.821333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11747.846891] Lustre: Skipped 1 previous similar message [11747.852997] LustreError: Skipped 1 previous similar message [11752.910302] Lustre: server umount lustre-OST0000 complete [11758.048707] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11758.051087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11758.060385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11758.060397] Lustre: Skipped 1 previous similar message [11758.079515] Lustre: Skipped 1 previous similar message [11763.176079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11763.185356] Lustre: Skipped 2 previous similar messages [11769.312895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11769.323549] Lustre: Skipped 3 previous similar messages [11771.359829] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [11771.618108] Lustre: server umount lustre-MDT0000 complete [11774.433821] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11774.455434] LustreError: Skipped 1 previous similar message [11775.510266] LustreError: 439087:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202755 with bad export cookie 15963916179284997230 [11775.517523] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11775.522062] LustreError: 439087:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [11775.728386] Lustre: server umount lustre-MDT0001 complete [11785.887719] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11787.788667] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 17:39:26 (1776202766) [11795.103724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11795.639403] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11795.742673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11800.114424] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [11801.057089] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11806.177500] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11808.641921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11809.092252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11813.446306] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [11821.349494] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11827.671686] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11834.547497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11834.858837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11838.938994] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [11846.694381] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11848.712631] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11851.307235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11859.424305] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11859.432197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11859.442794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11862.218938] Lustre: server umount lustre-OST0000 complete [11870.690200] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11870.692761] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11870.704451] LustreError: Skipped 1 previous similar message [11870.709198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11870.725336] Lustre: Skipped 2 previous similar messages [11870.758477] Lustre: Skipped 3 previous similar messages [11872.260057] Lustre: server umount lustre-MDT0000 complete [11875.760736] LustreError: 441585:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202855 with bad export cookie 15963916179284998084 [11875.765747] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11875.776241] LustreError: 441585:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [11886.652690] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [11889.175964] Key type lgssc unregistered [11889.579051] LNet: 445453:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11889.592956] LNet: Removed LNI 192.168.203.119@tcp [11890.490583] Key type .llcrypt unregistered [11890.494435] Key type ._llcrypt unregistered [11920.070973] alg: No test for adler32 (adler32-zlib) [11920.841731] Key type ._llcrypt registered [11920.845229] Key type .llcrypt registered [11920.965719] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [11921.262975] Key type .llcrypt unregistered [11921.266495] Key type ._llcrypt unregistered [11936.053376] alg: No test for adler32 (adler32-zlib) [11936.810546] Key type ._llcrypt registered [11936.814127] Key type .llcrypt registered [11936.921978] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [11937.825649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11938.288842] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [11938.468469] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [11938.475189] LNet: Accept secure, port 988 [11940.095960] Key type lgssc registered [11941.036756] Lustre: Echo OBD driver; http://www.lustre.org/ [11948.606714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11950.040190] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11950.079301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11954.854958] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [11955.180890] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11960.304721] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11962.850978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11963.124283] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11963.137481] LustreError: Skipped 1 previous similar message [11963.142883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11965.176257] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [11967.076535] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [11970.535689] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11970.551670] LustreError: Skipped 3 previous similar messages [11974.235581] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11980.772685] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11980.794054] LustreError: Skipped 3 previous similar messages [11980.858321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11981.160049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11984.967370] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [11995.267138] Lustre: server umount lustre-OST0000 complete [12000.752220] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12000.758300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12000.767960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12001.760569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12001.764823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12003.807182] Lustre: 447096:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202976/real 1776202976] req@00000000a3c3ff4c x1862483750957440/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1776202983 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [12003.835328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12003.845992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12003.850482] Lustre: Skipped 2 previous similar messages [12005.343151] Lustre: 450243:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202979/real 1776202979] req@0000000009461699 x1862483750958400/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776202985 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [12005.480380] Lustre: server umount lustre-MDT0000 complete [12008.554927] LustreError: 447763:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202988 with bad export cookie 10076781412746234599 [12008.559174] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12008.562187] LustreError: 447763:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [12008.818765] Lustre: server umount lustre-MDT0001 complete [12019.203211] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [12021.439171] Key type lgssc unregistered [12021.691164] LNet: 451047:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12021.703457] LNet: Removed LNI 192.168.203.119@tcp [12022.432598] Key type .llcrypt unregistered [12022.437194] Key type ._llcrypt unregistered [12044.556300] alg: No test for adler32 (adler32-zlib) [12045.318558] Key type ._llcrypt registered [12045.322189] Key type .llcrypt registered [12045.448593] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [12046.514983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12046.961969] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [12047.165345] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [12047.171513] LNet: Accept secure, port 988 [12048.831256] Key type lgssc registered [12050.025744] Lustre: Echo OBD driver; http://www.lustre.org/ [12057.599498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12074.207334] LustreError: 452484:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.119@tcp: failed processing log, type 1: rc = -5 [12105.951264] LustreError: 452484:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.119@tcp: failed processing log, type 4: rc = -110 [12135.711189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12140.200984] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [12148.319153] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12156.128769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12158.470214] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12158.566915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12158.627118] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [12163.287568] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [12163.563023] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12168.694745] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12171.891544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12172.203760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12177.032943] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [12184.857727] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12190.579306] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12197.856745] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12197.874861] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12197.896864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12199.074783] Lustre: server umount lustre-OST0000 complete [12202.981902] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12202.997143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12203.019094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12205.027232] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12205.031867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12205.050381] Lustre: Skipped 1 previous similar message [12208.095375] Lustre: 455373:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776203182/real 1776203182] req@000000009dd8a3b3 x1862483865254720/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776203188 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [12208.346754] Lustre: server umount lustre-MDT0000 complete [12210.144890] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12210.155498] LustreError: Skipped 1 previous similar message [12211.608718] LustreError: 452491:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203191 with bad export cookie 11663833068907795651 [12211.611718] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12211.619450] LustreError: 452491:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [12211.893819] Lustre: server umount lustre-MDT0001 complete [12223.186642] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [12225.952519] Key type lgssc unregistered [12226.266604] LNet: 456179:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12226.284527] LNet: Removed LNI 192.168.203.119@tcp [12227.068257] Key type .llcrypt unregistered [12227.074227] Key type ._llcrypt unregistered [12247.978738] alg: No test for adler32 (adler32-zlib) [12248.737846] Key type ._llcrypt registered [12248.740028] Key type .llcrypt registered [12248.836424] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [12249.732365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12250.140871] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [12250.328578] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [12250.331501] LNet: Accept secure, port 988 [12251.967119] Key type lgssc registered [12253.177286] Lustre: Echo OBD driver; http://www.lustre.org/ [12261.295233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12262.767910] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12262.823837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12267.200202] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [12268.003813] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12273.128216] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12276.071530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12276.551495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12281.054117] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [12289.101975] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12296.129255] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12303.314967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12303.629545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12308.714812] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [12312.882386] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [12316.627019] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12322.785260] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12322.795544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12322.809860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12323.297895] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12323.322256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12323.336434] Lustre: Skipped 2 previous similar messages [12326.428762] Lustre: server umount lustre-MDT0000 complete [12328.421337] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12328.435590] LustreError: Skipped 2 previous similar messages [12328.753371] LustreError: 457623:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.19@tcp arrived at 1776203308 with bad export cookie 14466216203907302519 [12330.858389] LustreError: 457623:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203310 with bad export cookie 14466216203907301623 [12330.866283] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12330.872415] LustreError: 457623:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [12332.518473] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12332.532995] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12332.558504] Lustre: Skipped 1 previous similar message [12332.562803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12332.570224] Lustre: Skipped 1 previous similar message [12333.536781] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12333.562187] LustreError: Skipped 1 previous similar message [12335.584693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12335.590673] Lustre: Skipped 1 previous similar message [12337.130269] Lustre: server umount lustre-OST0000 complete [12341.729459] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12341.762516] LustreError: Skipped 5 previous similar messages [12344.027353] Lustre: server umount lustre-MDT0001 complete [12356.783963] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [12359.747608] Key type lgssc unregistered [12360.067192] LNet: 461451:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12360.096284] LNet: Removed LNI 192.168.203.119@tcp [12360.971936] Key type .llcrypt unregistered [12360.974061] Key type ._llcrypt unregistered [12385.930859] alg: No test for adler32 (adler32-zlib) [12386.691558] Key type ._llcrypt registered [12386.694563] Key type .llcrypt registered [12386.832315] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [12388.064525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12388.897499] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [12389.127274] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [12389.130888] LNet: Accept secure, port 988 [12390.780114] Key type lgssc registered [12392.036708] Lustre: Echo OBD driver; http://www.lustre.org/ [12401.143953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12402.744513] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12402.843424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12407.813491] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [12408.292810] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12413.414117] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12417.158499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12417.588197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12422.211237] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [12430.626369] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12437.128558] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12444.208937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12444.461633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12448.522728] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [12453.681710] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:161 [12456.521704] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12463.233164] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.19@tcp (stopping) [12464.099132] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12464.112672] Lustre: Skipped 1 previous similar message [12464.119604] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12464.127540] Lustre: Skipped 1 previous similar message [12466.528519] Lustre: server umount lustre-MDT0001 complete [12468.311734] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12474.335922] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12474.337684] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12474.353555] LustreError: Skipped 2 previous similar messages [12474.376407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12479.449408] Lustre: server umount lustre-OST0000 complete [12489.696555] Lustre: 465964:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776203463/real 1776203463] req@000000008246562f x1862484223868416/t0(0) o251->MGC192.168.203.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776203469 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [12489.930627] Lustre: server umount lustre-MDT0000 complete [12504.622465] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [12507.692691] Key type lgssc unregistered [12507.962867] LNet: 466721:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12507.980629] LNet: Removed LNI 192.168.203.119@tcp [12508.790970] Key type .llcrypt unregistered [12508.793382] Key type ._llcrypt unregistered [12537.221535] alg: No test for adler32 (adler32-zlib) [12537.981832] Key type ._llcrypt registered [12537.984697] Key type .llcrypt registered [12538.093295] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [12551.521800] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [12552.800501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12553.293445] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [12553.520739] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [12553.527499] LNet: Accept secure, port 988 [12555.247177] Key type lgssc registered [12556.353534] Lustre: Echo OBD driver; http://www.lustre.org/ [12565.509784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12571.707920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12577.093971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12582.828353] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12590.550951] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [12601.359250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12608.353692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12608.456633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12609.704826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [12609.780899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [12609.900087] Lustre: lustre-MDT0000: new disk, initializing [12610.003750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12610.041412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12615.017721] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [12625.354226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12632.645936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12632.704946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12632.714180] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [12632.883395] Lustre: lustre-OST0000: new disk, initializing [12632.889829] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [12632.899612] Lustre: Skipped 1 previous similar message [12632.939976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12637.377413] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [12648.472760] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12656.350403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12662.308719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12662.369448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12662.458106] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [12662.519638] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [12662.611230] Lustre: lustre-MDT0001: new disk, initializing [12662.670056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12662.675567] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12662.744897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12667.066521] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [12676.388765] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [12676.613793] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12681.246565] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [12681.528144] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12687.843431] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12687.851773] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12687.852338] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12687.853751] Lustre: Skipped 1 previous similar message [12689.536756] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.19@tcp (stopping) [12689.543710] Lustre: Skipped 1 previous similar message [12692.633446] Lustre: server umount lustre-MDT0001 complete [12692.970182] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12692.986677] LustreError: Skipped 1 previous similar message [12694.612481] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12698.092734] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12698.105681] LustreError: Skipped 1 previous similar message [12703.212166] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12703.233907] LustreError: Skipped 1 previous similar message [12708.322304] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12708.346190] LustreError: Skipped 1 previous similar message [12711.904932] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12711.917368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12711.934429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12718.567408] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12718.571367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12718.591355] LustreError: Skipped 1 previous similar message [12718.601908] Lustre: Skipped 1 previous similar message [12723.680468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12725.218169] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [12725.451272] Lustre: server umount lustre-OST0000 complete [12741.087159] Lustre: 475134:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776203715/real 1776203715] req@000000004f383ebe x1862484395839104/t0(0) o251->MGC192.168.203.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776203721 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [12741.373306] Lustre: server umount lustre-MDT0000 complete [12757.596460] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [12760.403924] Key type lgssc unregistered [12760.766882] LNet: 475890:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12760.777391] LNet: Removed LNI 192.168.203.119@tcp [12761.641818] Key type .llcrypt unregistered [12761.644691] Key type ._llcrypt unregistered [12805.452822] alg: No test for adler32 (adler32-zlib) [12806.213131] Key type ._llcrypt registered [12806.218429] Key type .llcrypt registered [12806.319383] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [12806.570996] Key type .llcrypt unregistered [12806.573894] Key type ._llcrypt unregistered [12822.017697] alg: No test for adler32 (adler32-zlib) [12822.774473] Key type ._llcrypt registered [12822.778395] Key type .llcrypt registered [12822.914680] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [12824.165594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12824.780508] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [12825.016384] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [12825.020674] LNet: Accept secure, port 988 [12826.687158] Key type lgssc registered [12827.787937] Lustre: Echo OBD driver; http://www.lustre.org/ [12835.704360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12842.632911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12849.297042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12856.235523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12856.296652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12857.574643] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [12857.605771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [12857.703296] Lustre: lustre-MDT0000: new disk, initializing [12857.818052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12857.837654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12862.653200] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [12872.950557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12873.038588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12873.127404] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [12873.165410] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [12873.172152] Lustre: Skipped 1 previous similar message [12873.241616] Lustre: lustre-MDT0001: new disk, initializing [12873.380268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12873.418535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12873.423973] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12877.507234] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [12886.524740] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12891.704188] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12898.273299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12898.344914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12898.350453] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [12898.600420] Lustre: lustre-OST0000: new disk, initializing [12898.609856] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [12898.724610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12903.374343] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [12907.111416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12907.116827] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12912.888932] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12922.849213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12922.852317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12922.871234] Lustre: Skipped 1 previous similar message [12922.887570] Lustre: Skipped 2 previous similar messages [12925.649636] Lustre: server umount lustre-MDT0000 complete [12927.972995] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12927.988886] LustreError: Skipped 2 previous similar messages [12929.456783] LustreError: 479399:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203909 with bad export cookie 13872695348585885097 [12929.469870] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12929.473434] LustreError: 479399:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [12929.734330] Lustre: server umount lustre-MDT0001 complete [12939.747079] Lustre: 482657:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776203913/real 1776203913] req@00000000f07401c2 x1862484680000192/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776203919 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [12939.887324] Lustre: server umount lustre-OST0000 complete [12951.174206] Lustre: DEBUG MARKER: == conf-sanity test 73: failnode to update from mountdata properly ========================================================== 17:58:49 (1776203929) [12952.537276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12959.880303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12960.272526] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12960.346849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12965.270315] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [12965.345223] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12970.471211] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12974.349858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12974.792514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12980.015844] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [12988.021878] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12993.084491] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13000.227791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13000.340349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13000.669882] Lustre: Found index 0 for lustre-OST0000, updating log [13000.920511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13005.412952] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [13010.169414] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5 to 0x0:33 [13012.812978] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13020.641858] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13020.642497] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13020.645456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13020.669938] Lustre: Skipped 2 previous similar messages [13024.074449] Lustre: server umount lustre-OST0000 complete [13031.392237] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13031.401867] LustreError: Skipped 1 previous similar message [13031.407692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13031.424092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13031.433496] Lustre: Skipped 1 previous similar message [13034.305932] Lustre: server umount lustre-MDT0000 complete [13037.680056] LustreError: 483463:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204017 with bad export cookie 13872695348585886434 [13037.685739] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13037.689741] LustreError: 483463:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [13046.544202] Lustre: DEBUG MARKER: == conf-sanity test 75: The order of --index should be irrelevant ========================================================== 18:00:24 (1776204024) [13053.172109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13058.251381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13063.815685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13068.332601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13083.557586] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [13094.847344] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [13106.082611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13112.600789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13121.067933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13128.638986] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13136.589369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13136.689253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13136.988361] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [13137.016618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [13137.133139] Lustre: lustre-MDT0000: new disk, initializing [13137.226782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13137.252750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13142.313488] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [13153.117957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13153.197201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13153.343163] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [13153.405624] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [13153.409562] Lustre: Skipped 1 previous similar message [13153.500966] Lustre: lustre-MDT0001: new disk, initializing [13153.698907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13153.706577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13158.019312] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [13169.140437] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13175.498833] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13182.465594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13182.599088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13182.606233] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [13182.940226] Lustre: lustre-OST0000: new disk, initializing [13182.948666] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [13183.082529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13183.099641] Lustre: Skipped 1 previous similar message [13187.743987] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all 8 [13192.243753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13192.252525] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13197.758042] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13202.698387] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [13202.977479] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13207.535442] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [13207.785825] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13213.154607] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13213.157165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13213.175329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13213.175337] Lustre: Skipped 2 previous similar messages [13213.179069] LustreError: Skipped 1 previous similar message [13213.202365] Lustre: Skipped 2 previous similar messages [13215.751324] Lustre: server umount lustre-OST0000 complete [13215.756839] Lustre: Skipped 1 previous similar message [13220.326863] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13220.343940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13220.353685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13220.358916] Lustre: Skipped 1 previous similar message [13228.516909] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13228.537581] LustreError: Skipped 1 previous similar message [13229.836650] LustreError: 490518:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204209 with bad export cookie 13872695348585887505 [13229.837196] LustreError: 166-1: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13229.850750] LustreError: 490518:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [13251.603922] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [13260.047928] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [13269.581829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13278.543987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13285.464434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13291.883261] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13293.601664] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13055 sec ========= 18:04:32 (1776204272) [13333.957615] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [13337.161372] Key type lgssc unregistered [13337.405509] LNet: 499263:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13337.411491] LNet: Removed LNI 192.168.203.119@tcp [13338.158670] Key type .llcrypt unregistered [13338.163748] Key type ._llcrypt unregistered