[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 477557260 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002317] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008016] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.010124] LSM: Security Framework initializing [ 0.011033] Yama: becoming mindful. [ 0.012026] SELinux: Initializing. [ 0.013048] *** VALIDATE selinux *** [ 0.019632] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023166] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025091] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027099] *** VALIDATE tmpfs *** [ 0.029392] *** VALIDATE proc *** [ 0.030195] *** VALIDATE cgroup *** [ 0.031006] *** VALIDATE cgroup2 *** [ 0.033102] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034141] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036021] Spectre V2 : User space: Vulnerable [ 0.037006] Speculative Store Bypass: Vulnerable [ 0.038318] debug: unmapping init [mem 0xffffffff96e59000-0xffffffff96e60fff] [ 0.040000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040630] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041015] ... version: 2 [ 0.042009] ... bit width: 48 [ 0.043009] ... generic registers: 4 [ 0.044007] ... value mask: 0000ffffffffffff [ 0.045008] ... max period: 00007fffffffffff [ 0.046009] ... fixed-purpose events: 3 [ 0.046988] ... event mask: 000000070000000f [ 0.047261] rcu: Hierarchical SRCU implementation. [ 0.049371] smp: Bringing up secondary CPUs ... [ 0.050489] x86: Booting SMP configuration: [ 0.051014] .... node #0, CPUs: #1 #2 #3 [ 0.058317] smp: Brought up 1 node, 4 CPUs [ 0.060013] smpboot: Max logical packages: 1 [ 0.061009] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.114700] node 0 deferred pages initialised in 51ms [ 0.118140] devtmpfs: initialized [ 0.119402] x86/mm: Memory block size: 128MB [ 0.122466] gcov: version magic: 0x41383552 [ 0.124303] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.125098] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.126427] pinctrl core: initialized pinctrl subsystem [ 0.127208] [ 0.127515] ************************************************************* [ 0.128008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.129007] ** ** [ 0.130020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.131009] ** ** [ 0.132008] ** This means that this kernel is built to expose internal ** [ 0.133009] ** IOMMU data structures, which may compromise security on ** [ 0.134010] ** your system. ** [ 0.135007] ** ** [ 0.136008] ** If you see this message and you are not debugging the ** [ 0.137023] ** kernel, report this immediately to your vendor! ** [ 0.138006] ** ** [ 0.139007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.140008] ************************************************************* [ 0.141592] NET: Registered protocol family 16 [ 0.142309] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.143041] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.144039] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.145419] cpuidle: using governor menu [ 0.146678] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.149405] PCI: Using configuration type 1 for base access [ 0.152129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.159139] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.162026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.166023] cryptd: max_cpu_qlen set to 1000 [ 0.167235] ACPI: Added _OSI(Module Device) [ 0.168008] ACPI: Added _OSI(Processor Device) [ 0.169007] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.170011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.174261] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.176501] ACPI: Interpreter enabled [ 0.178047] ACPI: PM: (supports S0 S3 S4 S5) [ 0.179009] ACPI: Using IOAPIC for interrupt routing [ 0.180068] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.181329] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.190625] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.191022] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.192011] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.193064] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.195229] acpiphp: Slot [2] registered [ 0.196115] acpiphp: Slot [5] registered [ 0.197077] acpiphp: Slot [6] registered [ 0.198090] acpiphp: Slot [7] registered [ 0.199101] acpiphp: Slot [8] registered [ 0.200126] acpiphp: Slot [9] registered [ 0.201109] acpiphp: Slot [10] registered [ 0.202081] acpiphp: Slot [3] registered [ 0.203060] acpiphp: Slot [4] registered [ 0.204104] acpiphp: Slot [11] registered [ 0.205077] acpiphp: Slot [12] registered [ 0.206076] acpiphp: Slot [13] registered [ 0.207066] acpiphp: Slot [14] registered [ 0.208088] acpiphp: Slot [15] registered [ 0.209064] acpiphp: Slot [16] registered [ 0.210065] acpiphp: Slot [17] registered [ 0.211078] acpiphp: Slot [18] registered [ 0.212090] acpiphp: Slot [19] registered [ 0.213106] acpiphp: Slot [20] registered [ 0.214069] acpiphp: Slot [21] registered [ 0.216106] acpiphp: Slot [22] registered [ 0.218078] acpiphp: Slot [23] registered [ 0.219086] acpiphp: Slot [24] registered [ 0.220066] acpiphp: Slot [25] registered [ 0.221076] acpiphp: Slot [26] registered [ 0.222109] acpiphp: Slot [27] registered [ 0.224083] acpiphp: Slot [28] registered [ 0.225096] acpiphp: Slot [29] registered [ 0.227090] acpiphp: Slot [30] registered [ 0.229085] acpiphp: Slot [31] registered [ 0.231076] PCI host bridge to bus 0000:00 [ 0.233120] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.235031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.237020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.239029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.242026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.244025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.245172] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.248141] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.251374] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.260019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.264579] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.266030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.269022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.272016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.274390] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.276647] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.278031] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.280713] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.285016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.298013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.302012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.307175] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.311014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.317012] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.332020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.371916] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.381013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.390019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.423025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.453046] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.459017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.464018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.485016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.495056] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.501016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.509011] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.527012] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.536416] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.542016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.553016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.578016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.591488] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.597011] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.604015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.617013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.621264] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.624410] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.626884] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.630488] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.633245] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.637038] iommu: Default domain type: Passthrough [ 0.639967] SCSI subsystem initialized [ 0.641124] ACPI: bus type USB registered [ 0.643174] usbcore: registered new interface driver usbfs [ 0.644048] usbcore: registered new interface driver hub [ 0.645058] usbcore: registered new device driver usb [ 0.647359] pps_core: LinuxPPS API ver. 1 registered [ 0.650012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.651045] PTP clock support registered [ 0.654085] EDAC MC: Ver: 3.0.0 [ 0.656144] PCI: Using ACPI for IRQ routing [ 0.657738] NetLabel: Initializing [ 0.658008] NetLabel: domain hash size = 128 [ 0.659014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.660163] NetLabel: unlabeled traffic allowed by default [ 0.661186] vgaarb: loaded [ 0.662729] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.664018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.670608] clocksource: Switched to clocksource kvm-clock [ 0.774786] VFS: Disk quotas dquot_6.6.0 [ 0.776211] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.778248] *** VALIDATE ramfs *** [ 0.779312] *** VALIDATE hugetlbfs *** [ 0.781073] pnp: PnP ACPI init [ 0.782754] pnp: PnP ACPI: found 6 devices [ 0.799332] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.801968] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.804497] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.808056] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.811518] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.814154] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.816730] NET: Registered protocol family 2 [ 0.819239] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.824219] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.827366] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.832088] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.835990] TCP: Hash tables configured (established 65536 bind 65536) [ 0.838549] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.840466] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.842572] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.845242] NET: Registered protocol family 1 [ 0.847255] RPC: Registered named UNIX socket transport module. [ 0.848633] RPC: Registered udp transport module. [ 0.849602] RPC: Registered tcp transport module. [ 0.850619] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.852276] NET: Registered protocol family 44 [ 0.853401] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.854890] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.857193] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.860764] PCI: CLS 0 bytes, default 64 [ 0.862313] Unpacking initramfs... [ 2.568960] debug: unmapping init [mem 0xffff97bb7cc54000-0xffff97bb7ffbffff] [ 2.573163] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.575326] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.577973] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.101043] Initialise system trusted keyrings [ 3.102213] Key type blacklist registered [ 3.103494] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.111594] zbud: loaded [ 3.114015] *** VALIDATE nfs *** [ 3.114780] *** VALIDATE nfs4 *** [ 3.115968] pstore: using deflate compression [ 3.118913] Platform Keyring initialized [ 3.208673] NET: Registered protocol family 38 [ 3.210335] Key type asymmetric registered [ 3.211306] Asymmetric key parser 'x509' registered [ 3.212604] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.214872] io scheduler mq-deadline registered [ 3.215957] io scheduler kyber registered [ 3.217317] io scheduler bfq registered [ 3.218516] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.220730] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.222655] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.224842] ACPI: Power Button [PWRF] [ 3.229378] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.236022] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.254537] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.266855] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.288611] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.324955] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.360629] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.363706] Non-volatile memory driver v1.3 [ 3.364826] Linux agpgart interface v0.103 [ 3.406832] virtio_blk virtio1: [vda] 68000 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.410069] vda: detected capacity change from 0 to 34816000 [ 3.429160] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.432442] vdb: detected capacity change from 0 to 1073741824 [ 3.446598] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.449916] vdc: detected capacity change from 0 to 2621440000 [ 3.468062] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.472203] vdd: detected capacity change from 0 to 2621440000 [ 3.488266] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.491558] vde: detected capacity change from 0 to 4294967296 [ 3.514242] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.517841] vdf: detected capacity change from 0 to 4294967296 [ 3.524910] libphy: Fixed MDIO Bus: probed [ 3.532503] usbcore: registered new interface driver usbserial_generic [ 3.536492] usbserial: USB Serial support registered for generic [ 3.538944] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.545496] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.547303] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.551240] mousedev: PS/2 mouse device common for all mice [ 3.554136] rtc_cmos 00:05: RTC can wake from S4 [ 3.557472] rtc_cmos 00:05: registered as rtc0 [ 3.558881] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.561736] intel_pstate: CPU model not supported [ 3.562803] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.573947] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.574396] hid: raw HID events driver (C) Jiri Kosina [ 3.581738] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.584083] usbcore: registered new interface driver usbhid [ 3.588509] usbhid: USB HID core driver [ 3.590139] drop_monitor: Initializing network drop monitor service [ 3.592708] Initializing XFRM netlink socket [ 3.594768] NET: Registered protocol family 10 [ 3.600190] Segment Routing with IPv6 [ 3.601693] NET: Registered protocol family 17 [ 3.603735] mpls_gso: MPLS GSO support [ 3.609471] RAS: Correctable Errors collector initialized. [ 3.611658] AVX version of gcm_enc/dec engaged. [ 3.613328] AES CTR mode by8 optimization enabled [ 3.693421] sched_clock: Marking stable (3693400165, 0)->(4651468713, -958068548) [ 3.697231] registered taskstats version 1 [ 3.699327] Loading compiled-in X.509 certificates [ 3.701309] zswap: loaded using pool lzo/zbud [ 3.739802] Key type big_key registered [ 3.754538] Key type encrypted registered [ 3.756472] ima: No TPM chip found, activating TPM-bypass! [ 3.758854] ima: Allocated hash algorithm: sha1 [ 3.760799] ima: No architecture policies found [ 3.762494] evm: Initialising EVM extended attributes: [ 3.764314] evm: security.selinux [ 3.765487] evm: security.ima [ 3.766467] evm: security.capability [ 3.768188] evm: HMAC attrs: 0x1 [ 3.770371] rtc_cmos 00:05: setting system clock to 2026-02-24 21:28:44 UTC (1771968524) [ 3.777539] debug: unmapping init [mem 0xffffffff97e03000-0xffffffff97ffffff] [ 3.780938] debug: unmapping init [mem 0xffffffff96b82000-0xffffffff96e58fff] [ 3.789132] Write protecting the kernel read-only data: 28672k [ 3.793173] debug: unmapping init [mem 0xffffffff95203000-0xffffffff953fffff] [ 3.797117] debug: unmapping init [mem 0xffffffff95b14000-0xffffffff95bfffff] [ 3.836242] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.845431] systemd[1]: Detected virtualization kvm. [ 3.847616] systemd[1]: Detected architecture x86-64. [ 3.849553] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.878759] systemd[1]: No hostname configured. [ 3.881470] systemd[1]: Set hostname to . [ 3.884467] random: systemd: uninitialized urandom read (16 bytes read) [ 3.888232] systemd[1]: Initializing machine ID from random generator. [ 3.958933] random: ln: uninitialized urandom read (6 bytes read) [ 4.095294] random: systemd: uninitialized urandom read (16 bytes read) [ 4.097392] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.103494] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ 4.114455] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.926930] device-mapper: uevent: version 1.0.3 [ 4.929327] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 5.831772] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.119845] virtio_net virtio0 ens2: renamed from eth0 [ 6.396715] scsi host0: ata_piix [ 6.480825] scsi host1: ata_piix [ 6.482208] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.484554] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.296472] random: crng init done [ 11.301069] random: 7 urandom warning(s) missed due to ratelimiting [ 11.943518] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 12.699620] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.344827] printk: systemd: 26 output lines suppressed due to ratelimiting [ 14.850173] SELinux: Disabled at runtime. [ 14.946128] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.955354] systemd[1]: Detected virtualization kvm. [ 14.957387] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 15.877969] systemd[1]: initrd-switch-root.service: Succeeded. [ 15.881938] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 15.899401] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 15.905381] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.908469] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.920503] systemd[1]: Starting Journal Service... Starting Journal Service... [ 15.928513] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 16.182953] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Mounting Huge Pages File System... [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 17.323773] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.023843] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.338268] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 18.623760] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 18.772813] EDAC sbridge: Ver: 1.1.2 [ 21.209549] Key type dns_resolver registered [ 21.616470] NFS: Registering the id_resolver key type [ 21.618153] Key type id_resolver registered [ 21.619362] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg411-server login: [ 44.851604] libcfs: loading out-of-tree module taints kernel. [ 44.866521] alg: No test for adler32 (adler32-zlib) [ 45.617370] Key type ._llcrypt registered [ 45.618980] Key type .llcrypt registered [ 45.661498] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 52.375580] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 52.978958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 53.200763] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 53.442216] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 53.443956] LNet: Accept secure, port 988 [ 55.055102] Key type lgssc registered [ 55.505539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 61.788344] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 63.019206] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 66.597132] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 69.105515] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 71.914293] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 77.144424] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 81.449249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 81.483116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 82.565195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 82.574819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 82.611972] Lustre: lustre-MDT0000: new disk, initializing [ 82.640630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 82.648533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 84.091076] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 89.223050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 89.246762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 89.269650] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 89.278511] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 89.280383] Lustre: Skipped 1 previous similar message [ 89.320197] Lustre: lustre-MDT0001: new disk, initializing [ 89.343155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 89.351169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 89.354571] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 90.699049] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 95.847501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 95.873884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 95.878148] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 95.954532] Lustre: lustre-OST0000: new disk, initializing [ 95.956571] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 95.975781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 97.286332] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 99.561721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 99.564069] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 101.976819] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 102.004630] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 102.007740] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 102.037213] Lustre: lustre-OST0001: new disk, initializing [ 102.038720] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 102.052748] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 103.307076] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 105.705895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 105.709253] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 108.878115] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 115.595524] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 121.926555] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing check_logdir /tmp/testlogs/ [ 123.554569] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing yml_node [ 125.008291] Lustre: DEBUG MARKER: Client: 2.15.8.1 [ 125.850232] Lustre: DEBUG MARKER: MDS: 2.15.8.1 [ 126.697568] Lustre: DEBUG MARKER: OSS: 2.15.8.1 [ 127.305888] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Feb 24 16:30:47 EST 2026 [ 130.105309] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball 110 [ 130.643589] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 145.887636] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 145.889893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 145.896496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 146.911450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 146.911694] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 146.914374] Lustre: Skipped 2 previous similar messages [ 146.920049] Lustre: Skipped 2 previous similar messages [ 148.493399] Lustre: server umount lustre-MDT0000 complete [ 149.898810] LustreError: 6214:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771968670 with bad export cookie 14131923567062007862 [ 149.899794] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 149.903119] LustreError: 6214:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 152.031924] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 152.032958] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 152.033294] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 152.033299] Lustre: Skipped 1 previous similar message [ 152.037901] Lustre: Skipped 1 previous similar message [ 152.043330] LustreError: Skipped 1 previous similar message [ 154.079661] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 154.079714] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 154.082560] Lustre: Skipped 2 previous similar messages [ 154.086888] LustreError: Skipped 1 previous similar message [ 156.128052] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 156.132064] LustreError: Skipped 1 previous similar message [ 161.247527] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 161.247712] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 161.250218] Lustre: Skipped 3 previous similar messages [ 161.256373] LustreError: Skipped 1 previous similar message [ 164.319090] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 164.400127] Lustre: server umount lustre-MDT0001 complete [ 180.191149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 180.202346] Lustre: server umount lustre-OST0000 complete [ 187.666124] Lustre: server umount lustre-OST0001 complete [ 190.015748] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 193.190435] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 197.210118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 199.496172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 201.476913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 203.419193] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 205.657788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 205.679641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 205.759460] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 205.768297] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 205.800431] Lustre: lustre-MDT0000: new disk, initializing [ 205.821157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 205.827452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 207.188805] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 210.961214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 210.984354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 211.013046] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 211.091067] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 212.392128] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 215.938869] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 217.756982] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 219.771241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 219.792310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 219.795236] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 219.869622] Lustre: lustre-OST0000: new disk, initializing [ 219.871599] Lustre: Skipped 1 previous similar message [ 219.874085] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 219.876325] Lustre: Skipped 2 previous similar messages [ 219.898103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 219.900310] Lustre: Skipped 1 previous similar message [ 221.095474] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 221.298138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 221.301173] Lustre: Skipped 1 previous similar message [ 221.302613] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 224.641513] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 225.965363] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 226.037874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 227.337149] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 227.395994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 231.905258] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 231.905541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 231.907401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 231.909794] Lustre: Skipped 1 previous similar message [ 231.912480] Lustre: Skipped 1 previous similar message [ 234.064891] Lustre: server umount lustre-OST0000 complete [ 237.024355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 237.024671] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 237.027464] Lustre: Skipped 1 previous similar message [ 237.029426] LustreError: Skipped 1 previous similar message [ 242.147477] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 242.152760] LustreError: Skipped 1 previous similar message [ 242.721409] LustreError: 15163:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771968763 with bad export cookie 14131923567062009304 [ 242.721961] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 242.726790] LustreError: 15163:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 242.846035] Lustre: server umount lustre-MDT0001 complete [ 242.848155] Lustre: Skipped 1 previous similar message [ 244.131161] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 16:32:44 (1771968764) [ 244.952758] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 245.485945] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 16:32:45 (1771968765) [ 247.638446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 247.819599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 249.132195] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 251.813162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 253.182829] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 255.512299] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 257.227214] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 259.258539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 260.889209] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 263.492325] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 271.841312] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 271.841414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 271.844318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 271.847867] Lustre: Skipped 1 previous similar message [ 271.850518] Lustre: Skipped 3 previous similar messages [ 275.985581] Lustre: server umount lustre-OST0000 complete [ 277.471795] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 284.590108] LustreError: 18826:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771968805 with bad export cookie 14131923567062010158 [ 284.592446] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 284.593904] LustreError: 18826:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 289.321529] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 290.308675] Key type lgssc unregistered [ 290.421335] LNet: 22500:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 290.424140] LNet: Removed LNI 192.168.204.111@tcp [ 290.740422] Key type .llcrypt unregistered [ 290.741722] Key type ._llcrypt unregistered [ 299.333624] alg: No test for adler32 (adler32-zlib) [ 300.084362] Key type ._llcrypt registered [ 300.085415] Key type .llcrypt registered [ 300.118829] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 300.562497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 300.705131] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 300.774060] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 300.775383] LNet: Accept secure, port 988 [ 302.352882] Key type lgssc registered [ 302.668576] Lustre: Echo OBD driver; http://www.lustre.org/ [ 305.173864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 306.324304] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 306.339624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 307.920208] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 310.761239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 310.876742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 312.190385] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 314.527648] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 316.288491] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 318.262343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 318.361653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 319.596319] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 321.444632] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 321.822595] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 330.721039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 330.721458] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 330.721727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 330.725020] Lustre: Skipped 1 previous similar message [ 335.840778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 335.843511] Lustre: Skipped 2 previous similar messages [ 336.485277] Lustre: server umount lustre-OST0000 complete [ 340.960238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 340.960870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 340.963405] Lustre: Skipped 1 previous similar message [ 340.966617] Lustre: Skipped 1 previous similar message [ 346.079837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 346.081208] Lustre: Skipped 1 previous similar message [ 351.199841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 351.201893] Lustre: Skipped 1 previous similar message [ 352.223075] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 352.259497] Lustre: server umount lustre-MDT0000 complete [ 353.426062] LustreError: 23941:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771968874 with bad export cookie 7714780836909521937 [ 353.428890] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 353.430173] LustreError: 23941:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 353.541479] Lustre: server umount lustre-MDT0001 complete [ 357.839202] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 358.716163] Key type lgssc unregistered [ 358.814303] LNet: 27620:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 358.816380] LNet: Removed LNI 192.168.204.111@tcp [ 359.062682] Key type .llcrypt unregistered [ 359.064266] Key type ._llcrypt unregistered [ 367.169306] alg: No test for adler32 (adler32-zlib) [ 367.920285] Key type ._llcrypt registered [ 367.921064] Key type .llcrypt registered [ 367.955020] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 368.341656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 368.467371] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 368.536917] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 368.538242] LNet: Accept secure, port 988 [ 370.111094] Key type lgssc registered [ 370.433217] Lustre: Echo OBD driver; http://www.lustre.org/ [ 372.884703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 374.016089] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 374.029464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 375.355397] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 378.076939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 378.194188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 379.579913] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 381.970933] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 383.841423] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 385.992468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 386.084609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 387.358955] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 389.155751] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 389.594723] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 398.304823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 398.305848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 398.311117] Lustre: Skipped 2 previous similar messages [ 398.312812] Lustre: Skipped 1 previous similar message [ 400.887423] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.11@tcp (stopping) [ 400.890111] Lustre: Skipped 1 previous similar message [ 402.694290] Lustre: server umount lustre-MDT0000 complete [ 403.424831] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 403.430799] LustreError: Skipped 2 previous similar messages [ 403.943954] LustreError: 31725:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771968924 with bad export cookie 10964477317767472571 [ 403.945586] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 403.948024] LustreError: 31725:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 406.000881] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 406.008205] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.11@tcp (stopping) [ 408.543813] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 408.543985] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 408.553974] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 413.177804] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 413.177852] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.11@tcp (stopping) [ 413.182184] LustreError: Skipped 2 previous similar messages [ 413.184139] Lustre: Skipped 2 previous similar messages [ 418.271136] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 418.355221] Lustre: server umount lustre-MDT0001 complete [ 430.125219] Lustre: server umount lustre-OST0000 complete [ 436.264742] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 437.115678] Key type lgssc unregistered [ 437.213265] LNet: 33056:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 437.216911] LNet: Removed LNI 192.168.204.111@tcp [ 437.483083] Key type .llcrypt unregistered [ 437.484693] Key type ._llcrypt unregistered [ 445.644776] alg: No test for adler32 (adler32-zlib) [ 446.396286] Key type ._llcrypt registered [ 446.397055] Key type .llcrypt registered [ 446.430920] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 446.803192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 446.906295] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 446.965761] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 446.967109] LNet: Accept secure, port 988 [ 448.535092] Key type lgssc registered [ 448.833026] Lustre: Echo OBD driver; http://www.lustre.org/ [ 451.088336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 452.208063] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 452.221680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 453.459583] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 456.023627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 456.154085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 457.561181] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 459.945260] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 461.849585] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 464.090910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 464.186260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 465.453938] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 466.275540] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 467.691810] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 475.617967] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 475.618143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 475.618993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 475.629921] Lustre: Skipped 1 previous similar message [ 479.224780] Lustre: lustre-OST0000: Not available for connect from 192.168.204.11@tcp (stopping) [ 479.226474] Lustre: Skipped 1 previous similar message [ 480.737729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 480.739530] Lustre: Skipped 1 previous similar message [ 480.783921] Lustre: server umount lustre-OST0000 complete [ 496.096407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 496.097041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 496.100643] Lustre: Skipped 1 previous similar message [ 499.655479] Lustre: server umount lustre-MDT0000 complete [ 500.932311] LustreError: 34497:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771969021 with bad export cookie 864123993998293049 [ 500.934800] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 500.936592] LustreError: 34497:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 507.144252] Lustre: server umount lustre-MDT0001 complete [ 511.476524] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 512.319652] Key type lgssc unregistered [ 512.413325] LNet: 38341:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 512.416093] LNet: Removed LNI 192.168.204.111@tcp [ 512.659376] Key type .llcrypt unregistered [ 512.660259] Key type ._llcrypt unregistered [ 521.018921] alg: No test for adler32 (adler32-zlib) [ 521.770363] Key type ._llcrypt registered [ 521.771467] Key type .llcrypt registered [ 521.809858] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 522.135884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 522.384698] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 522.466421] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 522.467906] LNet: Accept secure, port 988 [ 524.047100] Key type lgssc registered [ 524.416105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 527.055885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 528.202043] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.216251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 529.687470] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 532.653531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 532.812540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 534.355369] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 536.914203] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 538.683728] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 540.824934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 540.919478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 542.181897] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 543.971554] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 544.378843] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 551.430971] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 552.572600] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 552.675051] LNetError: 120-3: Refusing connection from 192.168.204.111 for 127.0.0.2@tcp: No matching NI [ 552.677915] LNetError: 39104:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 552.681888] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 563.323315] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-02-2416h38m03s [ 563.860336] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 566.261079] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.11@tcp (stopping) [ 568.802158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 568.802213] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 568.802935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 568.807601] Lustre: Skipped 2 previous similar messages [ 568.809185] LNetError: 120-3: Refusing connection from 192.168.204.111 for 127.0.0.2@tcp: No matching NI [ 568.817546] LNetError: 39101:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 568.821585] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 570.633111] Lustre: server umount lustre-MDT0000 complete [ 571.375313] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 571.785428] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 573.920270] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.923036] LNetError: 120-3: Refusing connection from 192.168.204.111 for 127.0.0.2@tcp: No matching NI [ 573.924529] LustreError: Skipped 2 previous similar messages [ 573.927290] LNetError: 39102:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 573.933133] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 574.108360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 574.152854] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 574.153926] LustreError: 11-0: MGC192.168.204.111@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 574.156442] Lustre: Evicted from MGS (at 192.168.204.111@tcp) after server handle changed from 0x25f4ed0de8b76166 to 0x25f4ed0de8b765aa [ 574.162161] Lustre: MGC192.168.204.111@tcp: Connection restored to 192.168.204.111@tcp (at 0@lo) [ 574.246370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 574.275814] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 575.652789] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 576.766338] Lustre: DEBUG MARKER: Wait for df (28800) ... [ 577.291496] Lustre: DEBUG MARKER: done [ 579.039736] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 579.042380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.046425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 579.048996] Lustre: Skipped 2 previous similar messages [ 579.553909] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 579.554938] LNetError: 120-3: Refusing connection from 192.168.204.111 for 127.0.0.2@tcp: No matching NI [ 579.559519] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 579.562388] LNetError: 39103:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 579.566718] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 584.345380] Lustre: server umount lustre-OST0000 complete [ 589.792403] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 589.792648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 589.793132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 589.793137] Lustre: Skipped 1 previous similar message [ 589.795298] LustreError: Skipped 1 previous similar message [ 589.799111] Lustre: Skipped 2 previous similar messages [ 589.806057] LNetError: 120-3: Refusing connection from 192.168.204.111 for 127.0.0.2@tcp: No matching NI [ 589.808157] LNetError: 39104:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 589.810207] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 591.753700] Lustre: server umount lustre-MDT0000 complete [ 593.005238] LustreError: 39780:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771969113 with bad export cookie 2735071517715424682 [ 593.008909] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 593.011145] LustreError: 39780:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 597.864613] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 598.843347] Key type lgssc unregistered [ 598.956276] LNet: 44891:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 598.958646] LNet: Removed LNI 192.168.204.111@tcp [ 599.244704] Key type .llcrypt unregistered [ 599.245869] Key type ._llcrypt unregistered [ 610.147465] alg: No test for adler32 (adler32-zlib) [ 610.899333] Key type ._llcrypt registered [ 610.900160] Key type .llcrypt registered [ 610.934598] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 616.345678] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 616.685719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 616.793829] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 616.854347] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 616.855673] LNet: Accept secure, port 988 [ 618.431121] Key type lgssc registered [ 618.759282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 621.626474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 623.929405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 625.958271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 628.036465] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 630.257439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 630.282042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 631.365811] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 631.377321] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 631.415038] Lustre: lustre-MDT0000: new disk, initializing [ 631.437519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 631.443619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 632.761393] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 636.906394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 636.937584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 636.977381] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 636.996379] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 636.998422] Lustre: Skipped 1 previous similar message [ 637.034952] Lustre: lustre-MDT0001: new disk, initializing [ 637.067386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 637.079934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 637.083591] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 638.592612] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 642.243375] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 643.974567] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 646.015159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 646.035932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 646.039228] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 646.116034] Lustre: lustre-OST0000: new disk, initializing [ 646.117571] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 646.138903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 647.341824] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 647.857044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 647.861212] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 651.036073] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 652.475672] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 652.552727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 654.018982] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 654.094274] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 656.789954] Lustre: server umount lustre-OST0000 complete [ 658.400783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 658.401292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 658.410590] Lustre: Skipped 1 previous similar message [ 663.519114] Lustre: 52659:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771969178/real 1771969178] req@000000009c6dd8b0 x1858044309812096/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1771969184 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 663.520168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 663.528336] Lustre: Skipped 2 previous similar messages [ 665.567106] Lustre: 47207:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771969179/real 1771969179] req@00000000f45f9a4a x1858044309812288/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1771969186 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 665.567255] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 665.573654] Lustre: 47207:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 665.581106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 665.582867] Lustre: Skipped 1 previous similar message [ 665.604481] Lustre: server umount lustre-MDT0000 complete [ 666.820232] LustreError: 48972:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771969187 with bad export cookie 1097901875526647709 [ 666.824296] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 666.826620] LustreError: 48972:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 666.958633] Lustre: server umount lustre-MDT0001 complete [ 669.686104] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 16:39:49 (1771969189) [ 671.965039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 672.137777] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 672.161127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 673.691484] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 676.481775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 676.599509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 678.083833] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 680.646725] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 682.627382] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 684.866773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 684.974448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 686.375952] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 688.661676] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 696.015508] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-02-2416h40m16s [ 696.487324] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 698.636357] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 699.461389] Lustre: DEBUG MARKER: Creating a test file and stat it [ 699.491495] LustreError: 53528:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 sleeping for 40000ms [ 707.043324] LNetError: 120-3: Refusing connection from 192.168.204.111 for 127.0.0.2@tcp: No matching NI [ 707.045744] LNetError: 47196:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 707.048406] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 719.866106] Lustre: lustre-MDT0000: Client 97bd76ce-0897-4a7c-8926-dbcfff54492b (at 192.168.204.11@tcp) reconnecting [ 739.535101] LustreError: 53528:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 awake [ 740.255256] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 741.045447] Lustre: DEBUG MARKER: done [ 743.393638] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 743.394977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 743.402915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 743.405342] Lustre: Skipped 1 previous similar message [ 748.513808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 748.516569] Lustre: Skipped 1 previous similar message [ 748.690296] Lustre: server umount lustre-OST0000 complete [ 753.632458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 753.632582] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 753.635187] Lustre: Skipped 2 previous similar messages [ 753.639081] LNetError: 120-3: Refusing connection from 192.168.204.111 for 127.0.0.2@tcp: No matching NI [ 753.641405] LNetError: 47193:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 753.643334] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 758.751821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 758.753372] LNetError: 120-3: Refusing connection from 192.168.204.111 for 127.0.0.2@tcp: No matching NI [ 758.754377] Lustre: Skipped 3 previous similar messages [ 758.756455] LNetError: 47194:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 758.761053] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 763.872989] LNetError: 120-3: Refusing connection from 192.168.204.111 for 127.0.0.2@tcp: No matching NI [ 763.874903] LNetError: 47195:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 763.876831] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 764.383065] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 764.423133] Lustre: server umount lustre-MDT0000 complete [ 765.645180] LustreError: 56323:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771969286 with bad export cookie 1097901875526648549 [ 765.647791] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 765.648696] LustreError: 56323:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 770.071971] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 770.916569] Key type lgssc unregistered [ 771.007329] LNet: 57938:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 771.011230] LNet: Removed LNI 192.168.204.111@tcp [ 771.240033] Key type .llcrypt unregistered [ 771.240836] Key type ._llcrypt unregistered [ 781.218293] alg: No test for adler32 (adler32-zlib) [ 781.969322] Key type ._llcrypt registered [ 781.970859] Key type .llcrypt registered [ 782.006105] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 788.031156] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 788.392951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 788.602860] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 788.681446] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 788.682877] LNet: Accept secure, port 988 [ 790.263117] Key type lgssc registered [ 790.618025] Lustre: Echo OBD driver; http://www.lustre.org/ [ 793.480483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 795.599344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 797.564316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 799.524699] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 801.711481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 801.736511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 802.811261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 802.819771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 802.857444] Lustre: lustre-MDT0000: new disk, initializing [ 802.879641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 802.885639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 804.261784] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 808.389875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 808.415332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 808.450843] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 808.470083] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 808.472446] Lustre: Skipped 1 previous similar message [ 808.509862] Lustre: lustre-MDT0001: new disk, initializing [ 808.541591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 808.552405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 808.555584] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 809.941331] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 813.655972] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 815.522632] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 817.741719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 817.765869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 817.770369] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 817.859124] Lustre: lustre-OST0000: new disk, initializing [ 817.860949] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 817.885832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 819.202085] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 821.751936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 821.756950] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 822.935428] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 824.327945] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 824.400618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 825.782931] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 825.858875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 826.849616] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 826.850450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 826.852275] LustreError: Skipped 1 previous similar message [ 826.856486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 826.860432] Lustre: Skipped 1 previous similar message [ 828.564110] Lustre: server umount lustre-OST0000 complete [ 831.968461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 831.968897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 831.973899] Lustre: Skipped 2 previous similar messages [ 831.976795] Lustre: Skipped 1 previous similar message [ 837.088061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 837.090320] Lustre: Skipped 1 previous similar message [ 842.207854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 842.210566] Lustre: Skipped 1 previous similar message [ 844.255065] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 844.294535] Lustre: server umount lustre-MDT0000 complete [ 845.504885] LustreError: 64109:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771969366 with bad export cookie 12836835214815596749 [ 845.507191] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 845.508108] LustreError: 64109:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 845.625080] Lustre: server umount lustre-MDT0001 complete [ 848.450435] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 16:42:48 (1771969368) [ 848.960491] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 849.540645] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 16:42:49 (1771969369) [ 853.267467] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 854.752820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 854.897882] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 854.913734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 855.266209] Lustre: Failing over lustre-MDT0000 [ 855.267861] LustreError: 67039:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 855.271311] LustreError: 67039:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 855.274074] LustreError: 67039:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 861.663103] Lustre: 67090:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771969376/real 1771969376] req@00000000e825feec x1858044490170496/t0(0) o251->MGC192.168.204.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1771969382 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 861.733530] Lustre: server umount lustre-MDT0000 complete [ 863.771747] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 16:43:04 (1771969384) [ 865.922518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 866.059744] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 866.073473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 867.524214] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 870.373634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 871.895400] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 874.511388] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 876.502948] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 878.793774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 878.916693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 878.918885] Lustre: Skipped 1 previous similar message [ 880.306689] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 882.665051] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 889.307292] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 890.847498] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 890.849624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 890.852845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 890.854280] Lustre: Skipped 2 previous similar messages [ 896.960445] Lustre: server umount lustre-MDT0000 complete [ 898.084847] LustreError: 67638:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771969418 with bad export cookie 12836835214815597862 [ 898.087407] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 898.087855] LustreError: 67638:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 899.275126] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 900.722501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 903.356960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 903.648528] Lustre: Evicted from MGS (at 192.168.204.111@tcp) after server handle changed from 0xb225999d8b4f1526 to 0xb225999d8b4f1cb2 [ 903.652448] Lustre: MGC192.168.204.111@tcp: Connection restored to 192.168.204.111@tcp (at 0@lo) [ 903.744077] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 903.764260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 905.062950] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 906.784275] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 906.784898] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 906.787466] Lustre: Skipped 2 previous similar messages [ 906.792148] LustreError: Skipped 1 previous similar message [ 907.679617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 907.811136] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:65 [ 908.831141] Lustre: 60253:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771969422/real 1771969422] req@00000000239a0aee x1858044490184960/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1771969429 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 909.189435] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 910.880077] Lustre: 60253:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771969424/real 1771969424] req@000000005fa96435 x1858044490185024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1771969431 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 911.656589] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 912.867797] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 913.532850] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 917.984287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 917.985448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 917.985840] Lustre: Skipped 6 previous similar messages [ 917.989704] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 921.220513] Lustre: server umount lustre-MDT0000 complete [ 921.222308] Lustre: Skipped 1 previous similar message [ 922.375858] LustreError: 67638:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771969443 with bad export cookie 12836835214815599794 [ 922.378127] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 922.378454] LustreError: 67638:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 924.515201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 927.531427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 929.248407] Lustre: Evicted from MGS (at 192.168.204.111@tcp) after server handle changed from 0xb225999d8b4f1cb2 to 0xb225999d8b4f22af [ 929.252336] Lustre: MGC192.168.204.111@tcp: Connection restored to (at 0@lo) [ 929.346249] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 929.380822] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 930.399076] Lustre: 60252:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771969443/real 1771969443] req@0000000061ce8d6a x1858044490195136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1771969450 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 930.405972] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 930.409995] Lustre: Skipped 1 previous similar message [ 930.768198] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 933.561646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 933.720707] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:97 [ 934.495196] Lustre: 60255:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771969448/real 1771969448] req@00000000c12b7f04 x1858044490195264/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1771969455 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 935.215655] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 937.795494] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 938.979798] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 939.717717] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 944.097519] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 944.098037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 944.100919] Lustre: Skipped 2 previous similar messages [ 947.525815] Lustre: server umount lustre-MDT0000 complete [ 947.527770] Lustre: Skipped 1 previous similar message [ 948.192776] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 948.197379] LustreError: Skipped 4 previous similar messages [ 948.737125] LustreError: 67639:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771969469 with bad export cookie 12836835214815601327 [ 948.740285] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 948.742413] LustreError: 67639:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 950.938589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 952.005861] Lustre: DEBUG MARKER: files compared the same [ 958.943145] Lustre: 76778:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771969473/real 1771969473] req@00000000270c939b x1858044490204416/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1771969479 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 958.951191] Lustre: 76778:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 965.680276] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 966.661253] Key type lgssc unregistered [ 966.781334] LNet: 77685:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.783487] LNet: Removed LNI 192.168.204.111@tcp [ 967.061391] Key type .llcrypt unregistered [ 967.063858] Key type ._llcrypt unregistered [ 975.148429] alg: No test for adler32 (adler32-zlib) [ 975.899406] Key type ._llcrypt registered [ 975.900587] Key type .llcrypt registered [ 975.935201] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 976.316697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 976.597098] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 976.679593] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 976.681370] LNet: Accept secure, port 988 [ 978.263101] Key type lgssc registered [ 978.618079] Lustre: Echo OBD driver; http://www.lustre.org/ [ 981.166364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 982.320540] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 982.333209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 983.748212] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug +malloc all 8 [ 986.556112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 986.661334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 988.009832] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug +malloc all 8 [ 990.331452] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 992.142155] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 994.328514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 994.441337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 995.892759] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug +malloc all 8 [ 997.539404] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 998.435211] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1001.634207] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:129 [ 1007.072197] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1007.072482] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1007.072870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1007.080488] Lustre: Skipped 1 previous similar message [ 1012.124717] Lustre: server umount lustre-OST0000 complete [ 1017.311967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1017.311982] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1017.312262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1017.312269] Lustre: Skipped 1 previous similar message [ 1017.315539] Lustre: Skipped 1 previous similar message [ 1019.589402] Lustre: server umount lustre-MDT0000 complete [ 1020.845541] LustreError: 79128:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771969541 with bad export cookie 14855339273794457458 [ 1020.846968] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1020.848964] LustreError: 79128:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1020.959563] Lustre: server umount lustre-MDT0001 complete [ 1025.544523] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1026.597199] Key type lgssc unregistered [ 1026.718470] LNet: 82795:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.722151] LNet: Removed LNI 192.168.204.111@tcp [ 1027.053748] Key type .llcrypt unregistered [ 1027.054917] Key type ._llcrypt unregistered [ 1036.074079] alg: No test for adler32 (adler32-zlib) [ 1036.827522] Key type ._llcrypt registered [ 1036.829263] Key type .llcrypt registered [ 1036.865623] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1037.174942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1037.428148] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 1037.502585] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1037.504107] LNet: Accept secure, port 988 [ 1039.079098] Key type lgssc registered [ 1039.446732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1042.030217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1059.551264] LustreError: 84228:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 1: rc = -5 [ 1090.271238] LustreError: 84228:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 4: rc = -110 [ 1120.991161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1122.444130] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1124.912452] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1127.500203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1127.546816] Lustre: *** cfs_fail_loc=706, val=0*** [ 1142.586977] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1142.605759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1142.621651] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 1143.909223] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1146.567626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1146.680037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1146.722114] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:161 [ 1147.985982] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1150.230041] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1151.892029] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1152.991928] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1152.994429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1152.997809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1157.089203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1157.089980] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1157.093290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1158.627353] Lustre: server umount lustre-OST0000 complete [ 1162.208603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.209180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1162.211808] Lustre: Skipped 1 previous similar message [ 1162.212044] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1162.213847] Lustre: Skipped 2 previous similar messages [ 1166.022333] Lustre: server umount lustre-MDT0000 complete [ 1167.286967] LustreError: 84234:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771969688 with bad export cookie 6524183070046626208 [ 1167.289661] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1167.291817] LustreError: 84234:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1167.416818] Lustre: server umount lustre-MDT0001 complete [ 1171.173871] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1172.246841] Key type lgssc unregistered [ 1172.367476] LNet: 87964:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1172.371111] LNet: Removed LNI 192.168.204.111@tcp [ 1172.696799] Key type .llcrypt unregistered [ 1172.697856] Key type ._llcrypt unregistered [ 1181.730698] alg: No test for adler32 (adler32-zlib) [ 1182.481388] Key type ._llcrypt registered [ 1182.482486] Key type .llcrypt registered [ 1182.523332] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1182.856044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1183.132763] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 1183.216506] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1183.218243] LNet: Accept secure, port 988 [ 1184.799121] Key type lgssc registered [ 1185.172447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1187.868157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1190.298477] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1192.855587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1192.995717] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1193.027487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1194.382802] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1197.137185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1197.238749] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1197.242752] LustreError: Skipped 3 previous similar messages [ 1197.244511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1198.305832] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1198.311419] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:193 [ 1198.313440] LustreError: Skipped 2 previous similar messages [ 1198.625732] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1201.558881] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1201.563259] LustreError: Skipped 2 previous similar messages [ 1201.642178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1201.665220] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:193 [ 1203.055223] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1211.873533] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1211.875371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1211.875407] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1211.877725] Lustre: Skipped 1 previous similar message [ 1217.002571] Lustre: server umount lustre-OST0000 complete [ 1222.113103] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1222.113275] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1222.113988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1222.113997] Lustre: Skipped 1 previous similar message [ 1222.122901] Lustre: Skipped 1 previous similar message [ 1224.446912] Lustre: server umount lustre-MDT0000 complete [ 1225.853316] Lustre: server umount lustre-MDT0001 complete [ 1232.351177] Lustre: 92824:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771969747/real 1771969747] req@00000000b8d2a4bc x1858044903303808/t0(0) o251->MGC192.168.204.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1771969753 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1232.367568] Lustre: server umount MGS complete [ 1236.028494] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 16:49:16 (1771969756) [ 1245.615649] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 1248.445324] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1251.825566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1253.933913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1255.873191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1257.782065] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1259.779706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1259.801136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1261.081721] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1264.625836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1264.646437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1264.720546] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1264.732293] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1264.770862] Lustre: lustre-MDT0001: new disk, initializing [ 1264.790903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1264.799594] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1264.802168] LustreError: 97515:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osd get update log failed: rc = -115 [ 1266.031428] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1269.800361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1269.821648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1269.823961] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1269.904152] Lustre: lustre-OST0000: new disk, initializing [ 1269.906566] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1269.923565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1271.126506] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1274.872427] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1274.911588] Lustre: 98897:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@0000000040426104 x1858044903308224/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1771969807 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 1277.119024] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1277.135339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1277.187275] Lustre: lustre-MDT0000: new disk, initializing [ 1277.241855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1278.303812] Lustre: 99365:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@0000000018009ef5 x1858044903311232/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1771969811 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 1278.734686] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1283.362161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1283.365235] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1284.001121] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1303.007966] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1303.011018] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1303.015411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1303.017414] Lustre: Skipped 1 previous similar message [ 1308.640418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1308.642200] Lustre: Skipped 2 previous similar messages [ 1308.757490] Lustre: server umount lustre-OST0000 complete [ 1313.760936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1313.761413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1313.765757] Lustre: Skipped 2 previous similar messages [ 1313.769995] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1323.710803] Lustre: server umount lustre-MDT0001 complete [ 1323.712265] Lustre: Skipped 1 previous similar message [ 1331.167128] Lustre: 100562:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771969845/real 1771969845] req@000000007af55e77 x1858044903320704/t0(0) o251->MGC192.168.204.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1771969851 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1334.816711] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 16:50:55 (1771969855) [ 1341.710214] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1342.621650] Key type lgssc unregistered [ 1342.729374] LNet: 101967:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1342.732159] LNet: Removed LNI 192.168.204.111@tcp [ 1342.986887] Key type .llcrypt unregistered [ 1342.987928] Key type ._llcrypt unregistered [ 1349.648146] alg: No test for adler32 (adler32-zlib) [ 1350.400478] Key type ._llcrypt registered [ 1350.402481] Key type .llcrypt registered [ 1350.444231] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1350.909887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1351.092688] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 1351.175525] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1351.177100] LNet: Accept secure, port 988 [ 1352.759133] Key type lgssc registered [ 1353.072282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1355.594058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1356.737057] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1356.757277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1358.238407] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1361.160154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1361.287119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1362.829953] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1365.338764] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1367.255352] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1369.617984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1369.730311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1371.224988] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1372.839431] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 1373.827304] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1383.393446] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1383.394255] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1383.394458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1383.399645] Lustre: Skipped 1 previous similar message [ 1387.477712] Lustre: server umount lustre-OST0000 complete [ 1392.095855] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1392.098020] LustreError: Skipped 1 previous similar message [ 1392.099273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1392.102565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1392.104749] Lustre: Skipped 1 previous similar message [ 1393.631932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1393.632355] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1394.824440] Lustre: server umount lustre-MDT0000 complete [ 1396.059981] LustreError: 105189:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771969916 with bad export cookie 3022523274204846740 [ 1396.063076] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1396.064539] LustreError: 105189:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1396.184656] Lustre: server umount lustre-MDT0001 complete [ 1400.917873] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1401.904159] Key type lgssc unregistered [ 1402.016373] LNet: 106892:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1402.018874] LNet: Removed LNI 192.168.204.111@tcp [ 1402.310502] Key type .llcrypt unregistered [ 1402.312364] Key type ._llcrypt unregistered [ 1408.630391] alg: No test for adler32 (adler32-zlib) [ 1409.381417] Key type ._llcrypt registered [ 1409.383085] Key type .llcrypt registered [ 1409.420887] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1409.874953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1410.042557] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 1410.135392] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1410.137884] LNet: Accept secure, port 988 [ 1411.719074] Key type lgssc registered [ 1412.069108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1414.172880] LustreError: 108020:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 1414.173242] LustreError: 108019:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 1414.177944] LustreError: 108020:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4997 [ 1414.180319] LustreError: 108020:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1414.183959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1414.184628] LustreError: 108020:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1415.324925] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1415.339337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1417.834455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1417.927550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1419.375179] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1421.089579] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing lsmod [ 1422.075410] LustreError: 109328:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 1422.075481] LustreError: 109329:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 1422.081150] LustreError: 109328:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4997 [ 1422.084550] LustreError: 109328:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1422.087375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1422.090172] LustreError: 109328:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1422.202072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1423.209101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1423.210385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1423.214649] Lustre: Skipped 2 previous similar messages [ 1423.219011] Lustre: Skipped 2 previous similar messages [ 1428.448570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1428.450892] Lustre: Skipped 2 previous similar messages [ 1429.250261] Lustre: server umount lustre-MDT0000 complete [ 1430.537198] LustreError: 108032:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771969951 with bad export cookie 13675684154867946212 [ 1430.539458] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1430.540845] LustreError: 108032:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1430.677867] Lustre: server umount lustre-MDT0001 complete [ 1438.175204] Lustre: 109923:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771969952/real 1771969952] req@000000001ff13c27 x1858045141328064/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1771969958 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1438.241896] Lustre: server umount lustre-OST0000 complete [ 1440.985744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1441.137170] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1441.156685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1442.534723] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1445.340840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1445.467268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1446.949727] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1449.388595] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1451.276498] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1453.566154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1453.680867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1455.102500] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1456.739925] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 1457.365824] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1466.337679] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1466.337754] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1466.342456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1466.345957] Lustre: Skipped 1 previous similar message [ 1470.934199] Lustre: server umount lustre-OST0000 complete [ 1476.063874] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1476.066847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1476.071047] Lustre: Skipped 1 previous similar message [ 1476.073055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1478.279107] Lustre: server umount lustre-MDT0000 complete [ 1479.564125] LustreError: 113246:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771970000 with bad export cookie 13675684154867946898 [ 1479.566643] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1479.568114] LustreError: 113246:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1484.659661] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1485.641638] Key type lgssc unregistered [ 1485.732312] LNet: 114048:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1485.735321] LNet: Removed LNI 192.168.204.111@tcp [ 1485.985472] Key type .llcrypt unregistered [ 1485.986336] Key type ._llcrypt unregistered [ 1494.061685] alg: No test for adler32 (adler32-zlib) [ 1494.813317] Key type ._llcrypt registered [ 1494.814453] Key type .llcrypt registered [ 1494.847482] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1495.273647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1495.409743] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 1495.483060] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1495.484740] LNet: Accept secure, port 988 [ 1497.063106] Key type lgssc registered [ 1497.423622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1500.049748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1501.189329] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1501.205121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1502.683688] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1505.577868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1505.694149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1507.048336] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1509.570707] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1511.517715] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1513.794687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1513.905833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1515.335770] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1516.969093] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 1517.869147] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1524.618755] Lustre: Setting parameter lustre-client.llite.some_wrong_param in log lustre-client [ 1525.443567] Lustre: Modifying parameter lustre-client.llite.some_wrong_param in log lustre-client [ 1527.264117] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1527.267340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1527.271592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1531.361910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1531.362227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1531.362445] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1532.510641] Lustre: server umount lustre-OST0000 complete [ 1536.480470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1536.480716] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1536.481202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1536.481207] Lustre: Skipped 1 previous similar message [ 1536.483411] Lustre: Skipped 1 previous similar message [ 1541.599875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1541.601305] Lustre: Skipped 2 previous similar messages [ 1546.719978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1546.722104] Lustre: Skipped 1 previous similar message [ 1548.255115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1548.299295] Lustre: server umount lustre-MDT0000 complete [ 1549.552923] LustreError: 118108:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771970070 with bad export cookie 17190847251612917399 [ 1549.556100] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1549.556534] LustreError: 118108:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1549.682644] Lustre: server umount lustre-MDT0001 complete [ 1554.302640] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1555.236287] Key type lgssc unregistered [ 1555.347311] LNet: 119264:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1555.350141] LNet: Removed LNI 192.168.204.111@tcp [ 1555.630139] Key type .llcrypt unregistered [ 1555.631301] Key type ._llcrypt unregistered [ 1562.177733] alg: No test for adler32 (adler32-zlib) [ 1562.929426] Key type ._llcrypt registered [ 1562.930358] Key type .llcrypt registered [ 1562.968161] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1563.423985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1563.552075] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 1563.640923] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1563.643770] LNet: Accept secure, port 988 [ 1565.231135] Key type lgssc registered [ 1565.584382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1568.159695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1569.300435] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1569.315953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1570.713827] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1573.607939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1573.737123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1575.270490] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1577.736443] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1579.612422] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1581.842047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1581.944487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1583.319457] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1584.996093] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 1585.735865] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1594.337389] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1594.337976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1594.338663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1594.345900] Lustre: Skipped 1 previous similar message [ 1599.457589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1599.459524] Lustre: Skipped 2 previous similar messages [ 1599.509968] Lustre: server umount lustre-OST0000 complete [ 1604.577446] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1604.578029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1604.588579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1604.591025] Lustre: Skipped 1 previous similar message [ 1609.696405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1609.698334] Lustre: Skipped 1 previous similar message [ 1614.816235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1614.818638] Lustre: Skipped 1 previous similar message [ 1615.327113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1615.365226] Lustre: server umount lustre-MDT0000 complete [ 1616.693152] LustreError: 120510:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771970137 with bad export cookie 10630816010027752530 [ 1616.695974] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1616.699401] LustreError: 120510:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1616.841229] Lustre: server umount lustre-MDT0001 complete [ 1621.505956] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1622.510781] Key type lgssc unregistered [ 1622.624291] LNet: 124183:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1622.626806] LNet: Removed LNI 192.168.204.111@tcp [ 1622.914952] Key type .llcrypt unregistered [ 1622.916257] Key type ._llcrypt unregistered [ 1631.653134] alg: No test for adler32 (adler32-zlib) [ 1632.405358] Key type ._llcrypt registered [ 1632.406459] Key type .llcrypt registered [ 1632.442973] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1632.801727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1633.023798] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 1633.102819] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1633.104252] LNet: Accept secure, port 988 [ 1634.687080] Key type lgssc registered [ 1635.033509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1637.540248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1638.687747] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1638.702357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1640.099630] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1642.878285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1643.002674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1644.384409] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1646.889619] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1648.708724] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1650.993301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1651.101951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1652.484591] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1654.185523] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 1654.973221] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1662.331854] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 1662.404231] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 1663.573340] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 1663.576189] Lustre: Skipped 2 previous similar messages [ 1665.007831] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 1665.010637] Lustre: Skipped 2 previous similar messages [ 1669.116601] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1669.127181] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1669.128571] Lustre: Skipped 1 previous similar message [ 1671.229519] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 1671.232102] Lustre: Skipped 2 previous similar messages [ 1676.796157] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.11@tcp 192.168.0.[2,10]@tcp [ 1679.840232] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1679.841121] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1679.843015] LustreError: Skipped 1 previous similar message [ 1679.848498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1681.170710] Lustre: server umount lustre-OST0000 complete [ 1683.936724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1683.937254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1683.940050] Lustre: Skipped 1 previous similar message [ 1683.943207] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1688.513159] Lustre: server umount lustre-MDT0000 complete [ 1689.056056] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1689.060064] LustreError: Skipped 1 previous similar message [ 1689.742125] LustreError: 128341:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771970210 with bad export cookie 12636561497131540171 [ 1689.745083] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1689.745919] LustreError: 128341:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1696.223123] Lustre: 129899:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771970210/real 1771970210] req@0000000069f10351 x1858045375175104/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1771970216 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1696.345201] Lustre: server umount lustre-MDT0001 complete [ 1700.784413] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1701.663495] Key type lgssc unregistered [ 1701.766324] LNet: 130500:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1701.769215] LNet: Removed LNI 192.168.204.111@tcp [ 1702.040226] Key type .llcrypt unregistered [ 1702.041469] Key type ._llcrypt unregistered [ 1712.069612] alg: No test for adler32 (adler32-zlib) [ 1712.822380] Key type ._llcrypt registered [ 1712.823765] Key type .llcrypt registered [ 1712.860168] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1713.195277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1713.403334] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 1713.479707] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1713.481175] LNet: Accept secure, port 988 [ 1715.055130] Key type lgssc registered [ 1715.369228] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1717.938733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1719.076309] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1719.089415] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1719.093619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1719.097616] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.204.11@tcp 192.168.0.[2,10]@tcp [ 1720.492667] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1723.207842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1723.304158] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1723.312131] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.11@tcp 192.168.0.[2,10]@tcp [ 1723.326061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1724.647901] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1727.053307] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1728.929582] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1731.123873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1731.231106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1732.580139] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1734.311120] Lustre: lustre-OST0000: deleting orphan objects from 0x0:167 to 0x0:193 [ 1734.967742] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1743.842827] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1743.843876] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1743.845024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1743.846515] Lustre: Skipped 1 previous similar message [ 1748.888668] Lustre: server umount lustre-OST0000 complete [ 1754.080611] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1754.080655] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1754.081315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1754.081320] Lustre: Skipped 1 previous similar message [ 1754.084216] Lustre: Skipped 1 previous similar message [ 1756.231446] Lustre: server umount lustre-MDT0000 complete [ 1757.511994] LustreError: 132084:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771970278 with bad export cookie 9484308164420067245 [ 1757.514517] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1757.519416] LustreError: 132084:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1757.634404] Lustre: server umount lustre-MDT0001 complete [ 1762.225495] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1763.084153] Key type lgssc unregistered [ 1763.186270] LNet: 135810:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1763.188866] LNet: Removed LNI 192.168.204.111@tcp [ 1763.456381] Key type .llcrypt unregistered [ 1763.457362] Key type ._llcrypt unregistered [ 1772.646763] alg: No test for adler32 (adler32-zlib) [ 1773.398306] Key type ._llcrypt registered [ 1773.399293] Key type .llcrypt registered [ 1773.433235] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 1778.489716] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1778.924492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1779.060891] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 1779.140906] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1779.142700] LNet: Accept secure, port 988 [ 1780.719095] Key type lgssc registered [ 1781.035074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1783.836908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1785.953134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1787.929579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1790.062576] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1792.334975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1792.356500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1793.440186] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1793.450318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1793.490130] Lustre: lustre-MDT0000: new disk, initializing [ 1793.515411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1793.524121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1794.962843] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1799.113438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1799.138305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1799.171637] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1799.187490] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1799.189554] Lustre: Skipped 1 previous similar message [ 1799.228469] Lustre: lustre-MDT0001: new disk, initializing [ 1799.258978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1799.270212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1799.273812] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1800.698376] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1804.438379] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1806.313473] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1808.531492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1808.562319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1808.566420] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1808.673972] Lustre: lustre-OST0000: new disk, initializing [ 1808.675673] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1808.702266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1810.249443] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1811.765879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1811.768687] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1814.098377] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1815.502413] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1815.573812] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1816.936701] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1817.007139] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1819.734625] Lustre: server umount lustre-OST0000 complete [ 1822.176588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1822.177243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1822.181174] Lustre: Skipped 1 previous similar message [ 1822.184631] Lustre: Skipped 1 previous similar message [ 1826.271204] Lustre: 143002:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771970341/real 1771970341] req@0000000058737220 x1858045528257408/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1771970347 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1827.295934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1827.298096] Lustre: Skipped 1 previous similar message [ 1828.319149] Lustre: 137552:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771970342/real 1771970342] req@00000000cb0636f8 x1858045528257792/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1771970349 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1828.319265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1828.326604] Lustre: 137552:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1828.333521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1828.337267] Lustre: Skipped 1 previous similar message [ 1828.355758] Lustre: server umount lustre-MDT0000 complete [ 1829.584939] LustreError: 139316:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771970350 with bad export cookie 4995746728330054505 [ 1829.587191] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1829.588520] LustreError: 139316:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1829.703902] Lustre: server umount lustre-MDT0001 complete [ 1832.481939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1832.653331] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1832.676617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1834.140689] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1836.979916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1837.093632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1838.492775] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1841.037160] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1842.936950] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1845.189152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1845.297565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1846.670493] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1849.035397] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1850.466351] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1850.534636] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1851.872311] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1851.947634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1855.200193] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1855.218679] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1855.221278] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 1855.266400] Lustre: lustre-OST0001: new disk, initializing [ 1855.268156] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1856.597102] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1859.870247] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 1865.209409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1865.213091] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1866.069745] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 1867.430673] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 1867.498561] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1869.294576] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff91fb60035800.ost_server_uuid 40 [ 1869.761813] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff91fb60035800.ost_server_uuid in FULL state after 0 sec [ 1875.424252] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1875.425646] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1875.428484] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1875.430974] Lustre: Skipped 1 previous similar message [ 1877.547488] Lustre: server umount lustre-OST0001 complete [ 1880.546593] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1880.547140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1880.548077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1880.548083] Lustre: Skipped 1 previous similar message [ 1880.551768] LustreError: Skipped 1 previous similar message [ 1880.557626] Lustre: Skipped 1 previous similar message [ 1885.666058] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1885.669650] LustreError: Skipped 1 previous similar message [ 1890.784929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1890.785629] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1890.786542] Lustre: Skipped 4 previous similar messages [ 1890.792741] LustreError: Skipped 1 previous similar message [ 1893.343149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1893.401828] Lustre: server umount lustre-OST0000 complete [ 1895.904640] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1901.024895] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1901.030312] LustreError: Skipped 1 previous similar message [ 1901.948961] LustreError: 143656:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771970422 with bad export cookie 4995746728330055359 [ 1901.951222] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1901.952871] LustreError: 143656:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1902.073188] Lustre: server umount lustre-MDT0001 complete [ 1902.074541] Lustre: Skipped 1 previous similar message [ 1909.203246] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1910.131027] Key type lgssc unregistered [ 1910.243328] LNet: 150366:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1910.246316] LNet: Removed LNI 192.168.204.111@tcp [ 1910.516679] Key type .llcrypt unregistered [ 1910.517811] Key type ._llcrypt unregistered [ 1921.263552] alg: No test for adler32 (adler32-zlib) [ 1922.015262] Key type ._llcrypt registered [ 1922.016099] Key type .llcrypt registered [ 1922.051183] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 1927.630301] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1928.067892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1928.191178] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 1928.269957] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1928.271676] LNet: Accept secure, port 988 [ 1929.855078] Key type lgssc registered [ 1930.150481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1932.842395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1934.895871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1936.788809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1938.865395] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1941.184984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1941.210616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1942.294827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1942.304355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1942.348973] Lustre: lustre-MDT0000: new disk, initializing [ 1942.372057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1942.378909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1943.840134] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1948.025390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1948.047233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1948.077192] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1948.094351] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1948.096701] Lustre: Skipped 1 previous similar message [ 1948.141256] Lustre: lustre-MDT0001: new disk, initializing [ 1948.176050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1948.191624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1948.196984] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1949.587135] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1953.408079] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1955.273793] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1957.446307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1957.470417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1957.472983] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1957.554259] Lustre: lustre-OST0000: new disk, initializing [ 1957.556217] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1957.580535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1957.648045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1957.651458] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1958.864433] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 1962.636498] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1964.132880] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1964.216807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1965.633080] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1965.711177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1968.097091] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1968.097303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1968.099342] LustreError: Skipped 1 previous similar message [ 1968.099784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1968.103679] Lustre: Skipped 1 previous similar message [ 1968.107517] Lustre: Skipped 1 previous similar message [ 1972.434772] Lustre: server umount lustre-OST0000 complete [ 1973.728709] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1973.732062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1973.740577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1978.335798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1978.336241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.456144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1983.458753] Lustre: Skipped 2 previous similar messages [ 1988.063097] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1988.101120] Lustre: server umount lustre-MDT0000 complete [ 1988.575960] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1988.579929] LustreError: Skipped 1 previous similar message [ 1989.348116] LustreError: 155322:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771970510 with bad export cookie 13452632153395365200 [ 1989.350512] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1989.352557] LustreError: 155322:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1989.476701] Lustre: server umount lustre-MDT0001 complete [ 1992.200235] Lustre: DEBUG MARKER: == conf-sanity test 47: server restart does not make client loss lru_resize settings ========================================================== 17:01:52 (1771970512) [ 1997.390711] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 2000.075891] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2003.088310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2005.123325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2007.128659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2009.084148] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2011.307905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2011.329410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2011.412817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2011.421314] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2011.456091] Lustre: lustre-MDT0000: new disk, initializing [ 2011.476104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2011.481206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2012.710465] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2016.573710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2016.593742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2016.625175] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2016.641599] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2016.643734] Lustre: Skipped 1 previous similar message [ 2016.685565] Lustre: lustre-MDT0001: new disk, initializing [ 2016.714340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2016.726683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2016.731319] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2018.044729] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2021.616582] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2023.500247] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2025.718032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2025.738305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2025.741206] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2025.847650] Lustre: lustre-OST0000: new disk, initializing [ 2025.850525] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2025.877866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2026.272346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2026.277247] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2027.318303] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2031.065676] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2033.048988] Lustre: Failing over lustre-OST0000 [ 2033.130773] Lustre: server umount lustre-OST0000 complete [ 2043.871204] Lustre: 152681:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771970557/real 1771970557] req@0000000082a39117 x1858045684510080/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1771970564 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 2043.871273] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2043.877448] Lustre: 152681:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2043.880615] Lustre: Skipped 1 previous similar message [ 2046.170725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2046.275562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2046.283169] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2047.616704] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2048.328071] Lustre: 152681:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771970562/real 1771970562] req@0000000037db08b7 x1858045684511104/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1771970569 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2048.333802] Lustre: 152681:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2048.804608] Lustre: Failing over lustre-MDT0000 [ 2048.916313] Lustre: server umount lustre-MDT0000 complete [ 2049.283585] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2049.284257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2049.290041] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2049.290227] Lustre: Skipped 2 previous similar messages [ 2049.293581] LustreError: Skipped 1 previous similar message [ 2054.623878] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2054.627215] LustreError: Skipped 4 previous similar messages [ 2056.671188] Lustre: 152683:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771970570/real 1771970570] req@00000000e824bee0 x1858045684514304/t0(0) o400->MGC192.168.204.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1771970577 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2056.677634] Lustre: 152683:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2056.680176] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2056.683917] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2056.687708] LustreError: Skipped 3 previous similar messages [ 2061.791961] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2061.797810] LustreError: Skipped 3 previous similar messages [ 2062.013523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2062.817469] Lustre: Evicted from MGS (at 192.168.204.111@tcp) after server handle changed from 0xbab159ab1d210898 to 0xbab159ab1d210d37 [ 2062.823581] LustreError: 165764:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.204.111@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (000000008513f219) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 2062.831110] Lustre: MGC192.168.204.111@tcp: Connection restored to (at 0@lo) [ 2062.896922] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2062.943707] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2064.365089] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2067.936506] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2067.939877] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 2067.947188] Lustre: lustre-OST0000: Recovery over after 0:18, of 3 clients 3 recovered and 0 were evicted. [ 2067.973860] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2073.056585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2073.057338] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2073.062144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2073.064270] LustreError: Skipped 1 previous similar message [ 2073.067556] Lustre: Skipped 1 previous similar message [ 2075.097879] Lustre: server umount lustre-OST0000 complete [ 2078.176712] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2083.296923] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2083.301031] LustreError: Skipped 6 previous similar messages [ 2083.719368] LustreError: 161437:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771970604 with bad export cookie 13452632153395367223 [ 2083.722273] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2083.722723] LustreError: 161437:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2088.606655] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 2089.654761] Key type lgssc unregistered [ 2089.786314] LNet: 167290:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2089.789555] LNet: Removed LNI 192.168.204.111@tcp [ 2090.083619] Key type .llcrypt unregistered [ 2090.085226] Key type ._llcrypt unregistered [ 2099.018340] alg: No test for adler32 (adler32-zlib) [ 2099.769500] Key type ._llcrypt registered [ 2099.770464] Key type .llcrypt registered [ 2099.808492] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2100.137141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2100.384309] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 2100.461051] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 2100.463611] LNet: Accept secure, port 988 [ 2102.039084] Key type lgssc registered [ 2102.383298] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2105.046577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2106.182886] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2106.197722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2107.602491] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2110.352762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2110.469806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2111.837523] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2114.262363] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2116.180257] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2118.212396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2118.307410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2119.542379] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2121.381631] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 2121.867733] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2214.864214] Lustre: Failing over lustre-MDT0000 [ 2214.962407] Lustre: server umount lustre-MDT0000 complete [ 2215.416250] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2217.953342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2217.953722] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2217.954676] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2217.956521] Lustre: Skipped 2 previous similar messages [ 2220.528038] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2220.532329] LustreError: Skipped 2 previous similar messages [ 2225.119186] Lustre: 168062:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771970738/real 1771970738] req@00000000fa957599 x1858045866196288/t0(0) o400->MGC192.168.204.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1771970745 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2225.127372] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2225.132073] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2225.136277] LustreError: Skipped 6 previous similar messages [ 2228.100805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2231.266794] Lustre: Evicted from MGS (at 192.168.204.111@tcp) after server handle changed from 0x528707329cd001ad to 0x528707329cd26dc7 [ 2231.271669] Lustre: MGC192.168.204.111@tcp: Connection restored to 192.168.204.111@tcp (at 0@lo) [ 2231.344236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2231.377344] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2232.761580] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2233.839973] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2236.387822] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 2236.394217] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2236.410299] Lustre: lustre-OST0000: deleting orphan objects from 0x0:68 to 0x0:97 [ 2238.185243] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2238.720132] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2251.745079] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2251.745126] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2251.745732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2251.748652] Lustre: Skipped 1 previous similar message [ 2256.864947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2256.867298] Lustre: Skipped 4 previous similar messages [ 2257.542689] Lustre: server umount lustre-MDT0000 complete [ 2258.740918] LustreError: 168728:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771970779 with bad export cookie 5946729746926824903 [ 2258.743332] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2258.743866] LustreError: 168728:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2261.983850] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2261.984125] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2261.988626] LustreError: Skipped 8 previous similar messages [ 2261.994393] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2264.031940] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2270.176041] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2270.177872] Lustre: Skipped 1 previous similar message [ 2273.247097] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2273.333683] Lustre: server umount lustre-MDT0001 complete [ 2280.737492] Lustre: server umount lustre-OST0000 complete [ 2283.728372] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 2286.710713] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2289.819646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2291.883417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2293.754604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2295.759402] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2297.978758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2298.000364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2298.084714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2298.094650] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2298.134186] Lustre: lustre-MDT0000: new disk, initializing [ 2298.156173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2298.162418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2299.537705] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2303.604228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2303.628427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2303.660483] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2303.675944] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2303.678192] Lustre: Skipped 1 previous similar message [ 2303.717015] Lustre: lustre-MDT0001: new disk, initializing [ 2303.755092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2303.759480] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2305.115488] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2308.815488] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2310.625043] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2312.775632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2312.797022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2312.799370] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2312.884216] Lustre: lustre-OST0000: new disk, initializing [ 2312.886826] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2312.912545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2312.915061] Lustre: Skipped 1 previous similar message [ 2314.248490] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2317.903973] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2319.380524] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2320.883728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2320.886375] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2321.489449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2322.826610] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2322.903877] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2325.984984] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2325.985741] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2325.987396] LustreError: Skipped 1 previous similar message [ 2325.987800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2325.990818] Lustre: Skipped 1 previous similar message [ 2325.994949] Lustre: Skipped 1 previous similar message [ 2329.620516] Lustre: server umount lustre-OST0000 complete [ 2331.104329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2331.107575] Lustre: Skipped 1 previous similar message [ 2338.123908] LustreError: 176461:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771970858 with bad export cookie 5946729746926825624 [ 2338.126444] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2338.127449] LustreError: 176461:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2338.237963] Lustre: server umount lustre-MDT0001 complete [ 2338.239178] Lustre: Skipped 1 previous similar message [ 2340.927893] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:07:41 (1771970861) [ 2346.159903] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 2349.088778] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2352.203252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2354.227578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2356.058423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2357.939108] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2360.115788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2360.136163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2360.214045] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2360.224569] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2360.265429] Lustre: lustre-MDT0000: new disk, initializing [ 2360.289402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2360.296611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2361.682210] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2365.898910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2365.919616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2365.952166] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2365.966664] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2365.968470] Lustre: Skipped 1 previous similar message [ 2366.003751] Lustre: lustre-MDT0001: new disk, initializing [ 2366.039297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2366.042024] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2367.271808] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2370.818650] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2372.970135] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2375.181939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2375.209513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2375.213045] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2375.313108] Lustre: lustre-OST0000: new disk, initializing [ 2375.315506] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2376.764396] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2377.842224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2377.844891] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2380.456494] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2383.328655] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2383.329255] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2383.330452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2383.330458] Lustre: Skipped 4 previous similar messages [ 2383.331059] LustreError: Skipped 1 previous similar message [ 2383.334022] Lustre: Skipped 1 previous similar message [ 2388.430740] Lustre: server umount lustre-OST0000 complete [ 2391.519684] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2396.969179] LustreError: 182595:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771970917 with bad export cookie 5946729746926826464 [ 2396.971976] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2396.974578] LustreError: 182595:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2399.676691] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:08:39 (1771970919) [ 2404.706336] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 2407.408398] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2410.734148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2412.831687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2414.672896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2416.880548] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2419.195495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2419.219060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2419.317587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2419.328393] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2419.367202] Lustre: lustre-MDT0000: new disk, initializing [ 2419.392894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2419.396118] Lustre: Skipped 2 previous similar messages [ 2419.403504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2420.881148] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2425.159962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2425.185706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2425.314288] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2426.601807] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2430.115939] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2431.859568] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2433.986187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2434.011480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2434.013701] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2434.931650] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2435.324921] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2439.083114] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2444.769242] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2444.771760] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2444.774944] Lustre: Skipped 2 previous similar messages [ 2444.776788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2444.778404] Lustre: Skipped 4 previous similar messages [ 2447.059183] Lustre: server umount lustre-OST0000 complete [ 2447.060660] Lustre: Skipped 2 previous similar messages [ 2455.519926] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2455.524495] LustreError: Skipped 4 previous similar messages [ 2455.544978] LustreError: 191156:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771970976 with bad export cookie 5946729746926827528 [ 2455.547918] LustreError: 191156:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2455.548206] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2460.097863] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 2461.119574] Key type lgssc unregistered [ 2461.222395] LNet: 193012:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2461.225194] LNet: Removed LNI 192.168.204.111@tcp [ 2461.483044] Key type .llcrypt unregistered [ 2461.484275] Key type ._llcrypt unregistered [ 2469.982596] alg: No test for adler32 (adler32-zlib) [ 2470.733364] Key type ._llcrypt registered [ 2470.734286] Key type .llcrypt registered [ 2470.765848] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2471.049927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2471.267866] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 2471.332814] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 2471.334404] LNet: Accept secure, port 988 [ 2472.903097] Key type lgssc registered [ 2473.221878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2475.531639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2476.664610] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2476.678411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2477.932703] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2480.563489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2480.678258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2481.916278] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2484.091815] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2485.774329] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2487.824479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2487.919326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2489.116204] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2491.324949] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2492.004366] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2511.842360] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2511.844302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2511.849298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2511.855620] Lustre: Skipped 1 previous similar message [ 2511.862732] Lustre: Skipped 1 previous similar message [ 2515.349253] Lustre: server umount lustre-OST0000 complete [ 2521.568486] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2521.575254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2521.588220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2527.199818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2527.201543] Lustre: Skipped 3 previous similar messages [ 2531.295128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2531.335655] Lustre: server umount lustre-MDT0000 complete [ 2532.883655] LustreError: 195258:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771971053 with bad export cookie 18013701950334819327 [ 2532.888082] LustreError: 195258:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2532.888183] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2533.022046] Lustre: server umount lustre-MDT0001 complete [ 2538.501619] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 2539.339386] Key type lgssc unregistered [ 2539.448388] LNet: 198145:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2539.450887] LNet: Removed LNI 192.168.204.111@tcp [ 2539.764520] Key type .llcrypt unregistered [ 2539.765948] Key type ._llcrypt unregistered [ 2549.475970] alg: No test for adler32 (adler32-zlib) [ 2550.228314] Key type ._llcrypt registered [ 2550.229134] Key type .llcrypt registered [ 2550.259980] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2550.536639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2550.755061] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 2550.821242] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 2550.822759] LNet: Accept secure, port 988 [ 2552.399107] Key type lgssc registered [ 2552.878547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2555.947406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2557.095606] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2557.109909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2558.615166] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2562.530649] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2562.643807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2562.776177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2564.634947] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2568.390072] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2570.940068] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2573.856872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2573.973524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2575.813607] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2578.817456] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2588.329893] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 2593.768371] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2593.768578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2593.772289] LustreError: Skipped 1 previous similar message [ 2593.773135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2593.778993] Lustre: Skipped 1 previous similar message [ 2594.809627] Lustre: lustre-OST0000: Not available for connect from 192.168.204.11@tcp (stopping) [ 2594.812705] Lustre: Skipped 1 previous similar message [ 2598.528600] Lustre: server umount lustre-OST0000 complete [ 2601.514267] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff91fb429c3000.ost_server_uuid 40 [ 2602.077418] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2614.240322] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2614.243938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2614.249896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2615.266434] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2618.657607] Lustre: server umount lustre-MDT0000 complete [ 2620.387277] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2620.397619] LustreError: Skipped 1 previous similar message [ 2620.430933] LustreError: 199586:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771971141 with bad export cookie 12902209183222997227 [ 2620.435570] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2620.436827] LustreError: 199586:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2620.590901] Lustre: server umount lustre-MDT0001 complete [ 2624.255377] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 17:12:24 (1771971144) [ 2627.296259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2627.547866] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.587320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2629.403335] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2632.673099] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2633.482188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2633.685329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2635.631574] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2639.079475] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2641.415422] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2644.390155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2644.564055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2646.066599] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2649.146468] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2652.048973] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2652.086373] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2652.090316] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2652.167064] Lustre: lustre-OST0001: new disk, initializing [ 2652.170538] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2652.416178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2652.421054] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2653.873325] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2657.580104] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 2658.140309] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2662.884590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2662.891098] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2662.894115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2662.903122] Lustre: Skipped 3 previous similar messages [ 2665.458438] Lustre: server umount lustre-OST0000 complete [ 2668.006134] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2668.010562] LustreError: Skipped 1 previous similar message [ 2668.377062] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2668.486891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2670.392290] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2670.472333] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2681.318827] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2681.319415] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2681.324963] Lustre: Skipped 2 previous similar messages [ 2681.326636] Lustre: Skipped 1 previous similar message [ 2686.435993] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2686.445271] LustreError: Skipped 10 previous similar messages [ 2686.666681] Lustre: server umount lustre-OST0001 complete [ 2690.016205] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2690.022407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2690.031898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2690.034016] Lustre: Skipped 3 previous similar messages [ 2696.138510] LustreError: 203574:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771971216 with bad export cookie 12902209183222998291 [ 2696.141740] LustreError: 203574:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2696.142342] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2696.334732] Lustre: server umount lustre-MDT0001 complete [ 2696.338620] Lustre: Skipped 1 previous similar message [ 2698.847888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2700.330206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2701.612933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2702.941785] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2705.123698] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 17:13:45 (1771971225) [ 2707.401429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2707.527613] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2707.538804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2707.613146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2707.614900] Lustre: Skipped 1 previous similar message [ 2709.238794] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2712.595747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2712.623178] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2712.635521] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2714.341481] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2717.692844] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2719.631355] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2722.097159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2722.190969] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2724.123973] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2724.451566] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:129 [ 2727.318292] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2730.230651] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2730.271097] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2732.264797] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2732.395984] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3 to 0x0:33 [ 2735.657982] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2737.637108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2737.640371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2737.644119] Lustre: Skipped 2 previous similar messages [ 2737.647025] Lustre: Skipped 3 previous similar messages [ 2742.835735] Lustre: server umount lustre-OST0000 complete [ 2746.353594] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2758.118380] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2758.123040] LustreError: Skipped 1 previous similar message [ 2769.548905] LustreError: 209983:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771971290 with bad export cookie 12902209183222999628 [ 2769.556169] LustreError: 209983:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2769.556760] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2772.471503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2774.304183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2775.744530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2777.480952] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2779.648238] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 17:14:59 (1771971299) [ 2784.550070] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 2788.008599] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2791.744707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2794.590029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2796.473439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2798.904303] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2801.776573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2801.816379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2801.933910] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2801.943580] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2801.984782] Lustre: lustre-MDT0000: new disk, initializing [ 2802.006308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2802.008075] Lustre: Skipped 3 previous similar messages [ 2802.013453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2803.468160] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2808.359675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2808.381317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2808.407930] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2808.421104] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2808.422753] Lustre: Skipped 1 previous similar message [ 2808.457258] Lustre: lustre-MDT0001: new disk, initializing [ 2808.492650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2808.497210] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2809.906681] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2813.817335] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2815.969665] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2818.824568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2818.848897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2818.855083] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2818.991730] Lustre: lustre-OST0000: new disk, initializing [ 2818.994431] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2820.693626] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2825.692314] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2826.765171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2826.771592] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2827.919227] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2828.067097] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2829.897627] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2829.982418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2831.843531] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2831.844251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2831.847386] LustreError: Skipped 1 previous similar message [ 2831.852562] Lustre: Skipped 4 previous similar messages [ 2831.855165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2831.858287] Lustre: Skipped 8 previous similar messages [ 2836.931782] Lustre: server umount lustre-OST0000 complete [ 2836.934054] Lustre: Skipped 3 previous similar messages [ 2853.343284] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2854.999284] LustreError: 218185:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771971375 with bad export cookie 12902209183223001350 [ 2855.005752] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2855.007764] LustreError: 218185:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2858.689877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2858.863836] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2858.868494] LustreError: Skipped 9 previous similar messages [ 2860.532661] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2864.289795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2866.104554] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2868.849740] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2871.721614] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2874.639828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2874.782704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2874.785563] Lustre: Skipped 4 previous similar messages [ 2876.416246] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2879.640112] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2881.413816] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2889.773137] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 2891.356404] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2891.439889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2894.305383] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2894.309369] LustreError: Skipped 1 previous similar message [ 2900.561556] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2902.270077] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2926.362802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2927.522071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2927.525538] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2927.525602] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 2927.528377] Lustre: lustre-OST0000: deleting orphan objects from 0x0:34 to 0x0:65 [ 2927.534058] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 2927.534697] Lustre: Skipped 1 previous similar message [ 2927.923528] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2930.786882] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2934.460979] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 2937.824969] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2937.825067] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2937.826643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2937.826648] Lustre: Skipped 11 previous similar messages [ 2937.828337] Lustre: Skipped 5 previous similar messages [ 2937.830335] LustreError: Skipped 1 previous similar message [ 2941.458572] Lustre: server umount lustre-OST0000 complete [ 2941.459757] Lustre: Skipped 3 previous similar messages [ 2949.962836] LustreError: 227924:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771971470 with bad export cookie 12902209183223002190 [ 2949.965934] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2949.966101] LustreError: 227924:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2952.879785] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 17:17:53 (1771971473) [ 2955.338323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2957.006011] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2959.776414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2961.580627] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2964.420567] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2966.854278] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2969.564582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2971.508801] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2974.731341] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2976.427588] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2976.495844] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2978.117555] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2978.207698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2981.085298] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2981.123865] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2981.128897] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2981.196281] Lustre: lustre-OST0001: new disk, initializing [ 2981.198937] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2982.980839] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 2987.808496] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2989.055735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2989.058949] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2989.597101] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 2989.724363] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2991.689402] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 2991.826350] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2999.266367] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2999.270241] LustreError: Skipped 3 previous similar messages [ 3000.770821] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3000.877143] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3002.522625] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 3002.612760] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3026.856298] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3026.946639] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3026.951139] Lustre: Skipped 5 previous similar messages [ 3028.598473] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3031.751471] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3032.041393] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3032.049401] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3032.049900] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to (at 0@lo) [ 3032.062239] Lustre: Skipped 1 previous similar message [ 3032.325469] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3037.156374] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3037.162387] LustreError: Skipped 3 previous similar messages [ 3062.975173] LustreError: 232329:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771971583 with bad export cookie 12902209183223003583 [ 3062.977916] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3062.982111] LustreError: 232329:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3071.401540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3073.086674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3074.547648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3076.161997] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3078.670219] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 17:19:58 (1771971598) [ 3081.105428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3081.222824] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3081.241155] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3082.939189] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3086.232113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3086.271501] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3088.264798] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3091.165444] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3093.658518] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3096.550647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3096.662454] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3098.120687] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3101.397446] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3111.218673] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3111.251207] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3113.282788] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3116.430461] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3117.103211] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3118.837877] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3118.970558] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3120.991966] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 3121.100845] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3123.741599] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff91fb51314000.ost_server_uuid 40 [ 3124.496071] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff91fb51314000.ost_server_uuid in FULL state after 0 sec [ 3125.005201] Lustre: Permanently deactivating lustre-OST0001 [ 3125.012533] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 3125.014851] Lustre: Skipped 1 previous similar message [ 3125.966448] Lustre: Permanently reactivating lustre-OST0001 [ 3128.806162] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3128.807891] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3128.815072] Lustre: Skipped 11 previous similar messages [ 3128.821433] Lustre: Skipped 15 previous similar messages [ 3133.363433] Lustre: server umount lustre-OST0001 complete [ 3133.366524] Lustre: Skipped 7 previous similar messages [ 3150.878891] LustreError: 237690:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771971671 with bad export cookie 12902209183223004843 [ 3150.883484] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3150.888705] LustreError: 237690:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3153.895843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3155.704745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3157.237712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3159.061907] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3161.989468] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 17:21:22 (1771971682) [ 3162.425254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3164.789969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3164.886676] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3164.897919] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3164.900441] Lustre: Skipped 5 previous similar messages [ 3166.649370] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3170.253817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3170.314053] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3172.424330] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3175.715950] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3178.426904] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3181.487534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3181.524715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3181.642671] Lustre: Permanently deactivating lustre-OST0000 [ 3183.373961] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3186.692325] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3189.233737] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3189.525886] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3191.183316] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3194.141791] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3199.512888] Lustre: lustre-OST0001: deleting orphan objects from 0x0:67 to 0x0:97 [ 3205.142614] Lustre: Permanently reactivating lustre-OST0000 [ 3205.150492] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 3205.153377] Lustre: Skipped 4 previous similar messages [ 3208.888293] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3208.921626] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 3214.827030] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3214.831517] LustreError: Skipped 8 previous similar messages [ 3228.048867] LustreError: 244528:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771971748 with bad export cookie 12902209183223006432 [ 3228.053533] LustreError: 244528:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3228.053569] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3233.765750] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 3234.716939] Key type lgssc unregistered [ 3234.825367] LNet: 249494:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3234.828769] LNet: Removed LNI 192.168.204.111@tcp [ 3235.117328] Key type .llcrypt unregistered [ 3235.118614] Key type ._llcrypt unregistered [ 3244.390524] alg: No test for adler32 (adler32-zlib) [ 3245.142294] Key type ._llcrypt registered [ 3245.143285] Key type .llcrypt registered [ 3245.175971] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3245.454677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3245.674996] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 3245.746031] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 3245.747682] LNet: Accept secure, port 988 [ 3247.327102] Key type lgssc registered [ 3247.875887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3249.734139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3252.946216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3254.143255] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3254.165982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3255.814263] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3258.786846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3258.823500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3258.862040] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3258.873116] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3258.877394] Lustre: Permanently deactivating lustre-MDT0001 [ 3259.007248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3260.510062] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3261.994841] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3263.597130] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3265.996206] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3268.410192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3268.536408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3269.537439] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3269.540736] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 3270.164755] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3272.728167] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3273.633933] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3273.640065] LustreError: Skipped 1 previous similar message [ 3273.640203] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:163 to 0x280000400:193 [ 3275.350115] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3275.414803] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3275.789538] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:99 to 0x2c0000400:129 [ 3275.791172] Lustre: lustre-OST0001: deleting orphan objects from 0x0:67 to 0x0:129 [ 3277.041962] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3279.658586] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3281.441936] Lustre: Permanently reactivating lustre-MDT0001 [ 3281.452843] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 3281.458594] Lustre: Skipped 3 previous similar messages [ 3287.062526] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3287.071107] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 3287.076513] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 3287.084944] LustreError: 254610:0:(llog_osd.c:2133:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 3287.085944] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 192.168.204.111@tcp (at 0@lo) [ 3287.091106] LustreError: 254610:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 3287.101885] LustreError: 254610:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.204.111@tcp: cfg command failed: rc = -108 [ 3287.107887] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 3287.107887] [ 3287.113764] LustreError: 250996:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 3290.110122] Lustre: lustre-MDT0001: Client 64d8540b-2484-475e-9ab1-cef4759b9cf4 (at 192.168.204.11@tcp) reconnecting [ 3293.299185] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3293.883513] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3295.773811] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 3295.868063] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3297.731650] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 3297.832403] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3298.808295] Lustre: Permanently deactivating lustre-MDT0001 [ 3298.818705] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 3298.822379] Lustre: Skipped 2 previous similar messages [ 3306.500643] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3315.679685] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3315.683264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3315.689872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3316.703973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3316.705276] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3316.710616] Lustre: Skipped 2 previous similar messages [ 3318.222269] Lustre: server umount lustre-MDT0000 complete [ 3319.532309] LustreError: 250986:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771971840 with bad export cookie 8368214703171070576 [ 3319.536716] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3319.537295] LustreError: 250986:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3319.678392] Lustre: server umount lustre-MDT0001 complete [ 3327.337721] Lustre: server umount lustre-OST0000 complete [ 3331.921095] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 17:24:12 (1771971852) [ 3336.433470] Lustre: server umount lustre-OST0001 complete [ 3339.118639] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 3343.042298] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3347.006926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3350.112194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3352.793580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3355.731757] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3358.751212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3358.797107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3358.918662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3358.920652] Lustre: Skipped 2 previous similar messages [ 3358.929287] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3358.973636] Lustre: lustre-MDT0000: new disk, initializing [ 3358.999106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3359.007213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3360.737427] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3366.038857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3366.085655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3366.146124] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3366.177167] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3366.180063] Lustre: Skipped 1 previous similar message [ 3366.229715] Lustre: lustre-MDT0001: new disk, initializing [ 3366.284618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3366.288324] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3368.168220] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3373.245124] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3375.633632] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3378.841757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3378.879606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3378.883726] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3379.020817] Lustre: lustre-OST0000: new disk, initializing [ 3379.024722] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3379.066593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3379.070110] Lustre: Skipped 1 previous similar message [ 3380.546506] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3385.360900] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3386.878928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3386.883590] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3391.720979] LustreError: 259541:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3396.755980] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3396.788823] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3396.792340] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3396.849225] Lustre: lustre-OST0001: new disk, initializing [ 3396.851249] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3396.887773] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3398.470146] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3401.815072] LustreError: 259541:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 3401.823183] LustreError: 259541:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3403.239789] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3404.825179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3404.830948] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3405.987581] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 3411.911102] LustreError: 259541:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 3415.011618] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3415.012772] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3415.015920] LustreError: Skipped 1 previous similar message [ 3415.019887] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3415.023559] Lustre: Skipped 4 previous similar messages [ 3420.132619] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3420.137130] Lustre: Skipped 2 previous similar messages [ 3425.254784] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3425.258881] Lustre: Skipped 2 previous similar messages [ 3427.295166] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3427.317147] Lustre: server umount lustre-OST0001 complete [ 3430.369949] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3430.373190] LustreError: Skipped 1 previous similar message [ 3435.490070] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3435.493181] LustreError: Skipped 1 previous similar message [ 3440.612830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3440.614663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3440.621687] Lustre: Skipped 2 previous similar messages [ 3440.626065] Lustre: Skipped 1 previous similar message [ 3445.731768] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3445.738685] LustreError: Skipped 3 previous similar messages [ 3445.798533] Lustre: server umount lustre-OST0000 complete [ 3448.288145] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3450.850187] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3450.861140] Lustre: Skipped 1 previous similar message [ 3454.677791] LustreError: 259528:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771971975 with bad export cookie 8368214703171073348 [ 3454.681223] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3454.684516] LustreError: 259528:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3466.625174] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 3467.743470] Key type lgssc unregistered [ 3467.870380] LNet: 264736:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3467.876715] LNet: Removed LNI 192.168.204.111@tcp [ 3468.288311] Key type .llcrypt unregistered [ 3468.290364] Key type ._llcrypt unregistered [ 3482.788031] alg: No test for adler32 (adler32-zlib) [ 3483.539297] Key type ._llcrypt registered [ 3483.540263] Key type .llcrypt registered [ 3483.574970] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 3491.066947] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3491.370647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3491.591453] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 3491.659050] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 3491.660463] LNet: Accept secure, port 988 [ 3493.239127] Key type lgssc registered [ 3493.789795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3498.193995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3501.455660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3504.242645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3507.542621] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3510.731139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3510.773821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3511.879644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3511.894766] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3511.931822] Lustre: lustre-MDT0000: new disk, initializing [ 3511.972485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3511.981769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3514.082755] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3518.969257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3518.989055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3519.019553] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3519.034525] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3519.036363] Lustre: Skipped 1 previous similar message [ 3519.073491] Lustre: lustre-MDT0001: new disk, initializing [ 3519.101642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3519.113414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3519.117352] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3520.917688] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3525.287444] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3527.768672] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3530.989957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3531.030729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3531.036243] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3531.157843] Lustre: lustre-OST0000: new disk, initializing [ 3531.160505] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3531.191146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3532.801721] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3536.980499] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3538.749206] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3540.480859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3540.486995] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3540.962808] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3542.508223] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3542.647541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3545.571337] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3545.572710] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3545.573460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3545.578762] Lustre: Skipped 1 previous similar message [ 3545.591614] LustreError: Skipped 1 previous similar message [ 3549.744297] Lustre: server umount lustre-OST0000 complete [ 3555.296433] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3555.301823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3555.310860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3555.314843] Lustre: Skipped 1 previous similar message [ 3557.712325] Lustre: server umount lustre-MDT0000 complete [ 3559.222341] LustreError: 268821:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771972079 with bad export cookie 15196849842935392435 [ 3559.226070] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3559.227543] LustreError: 268821:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3559.350593] Lustre: server umount lustre-MDT0001 complete [ 3562.612571] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 17:28:02 (1771972082) [ 3565.684163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3565.913146] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3565.954437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3567.829088] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3571.169677] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3571.881449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3572.053235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3573.279829] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3576.688754] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3578.946987] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3581.929409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3582.042792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3583.940494] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3587.688407] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3605.986438] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3605.989141] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3605.993971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3606.001581] Lustre: Skipped 2 previous similar messages [ 3606.008723] Lustre: Skipped 2 previous similar messages [ 3610.109284] Lustre: server umount lustre-OST0000 complete [ 3612.220202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3617.593904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3617.768132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3619.317465] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3621.861961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3621.867573] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3621.867727] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 3621.870665] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 3621.877829] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 3621.878167] Lustre: Skipped 1 previous similar message [ 3622.472743] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3626.980295] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3626.981267] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3626.988840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3626.996466] Lustre: Skipped 1 previous similar message [ 3631.925820] Lustre: server umount lustre-OST0000 complete [ 3633.631950] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3633.634862] LustreError: Skipped 1 previous similar message [ 3633.636165] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3633.639299] Lustre: Skipped 1 previous similar message [ 3633.640723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3641.352839] LustreError: 276029:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771972162 with bad export cookie 15196849842935393275 [ 3641.355566] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3641.355858] LustreError: 276029:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3641.472449] Lustre: server umount lustre-MDT0001 complete [ 3641.473801] Lustre: Skipped 1 previous similar message [ 3646.901534] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 3648.261449] Key type lgssc unregistered [ 3648.391621] LNet: 278825:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3648.395919] LNet: Removed LNI 192.168.204.111@tcp [ 3648.788226] Key type .llcrypt unregistered [ 3648.789371] Key type ._llcrypt unregistered [ 3658.842197] alg: No test for adler32 (adler32-zlib) [ 3659.594331] Key type ._llcrypt registered [ 3659.595298] Key type .llcrypt registered [ 3659.638452] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3659.988515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3660.166982] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 3660.233299] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 3660.234840] LNet: Accept secure, port 988 [ 3661.807077] Key type lgssc registered [ 3662.101094] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3665.052131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3666.240237] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3666.264341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3668.008672] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3671.525353] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3671.984466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3672.100595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3673.910479] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3676.965556] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3679.628726] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3682.543367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3682.701453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3684.258976] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3687.609619] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3698.086700] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:97 [ 3708.390053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3708.390441] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3708.390980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3708.394341] Lustre: Skipped 1 previous similar message [ 3709.738363] Lustre: server umount lustre-OST0000 complete [ 3712.992409] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3712.998132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3713.006889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3713.011262] Lustre: Skipped 1 previous similar message [ 3717.544921] Lustre: server umount lustre-MDT0000 complete [ 3718.625830] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3718.634214] LustreError: Skipped 1 previous similar message [ 3719.499379] LustreError: 281069:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771972240 with bad export cookie 15354892701189156516 [ 3719.505958] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3719.507596] LustreError: 281069:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3719.699199] Lustre: server umount lustre-MDT0001 complete [ 3725.812219] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 3727.004093] Key type lgssc unregistered [ 3727.123455] LNet: 284729:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3727.127179] LNet: Removed LNI 192.168.204.111@tcp [ 3727.456216] Key type .llcrypt unregistered [ 3727.457739] Key type ._llcrypt unregistered [ 3735.223638] alg: No test for adler32 (adler32-zlib) [ 3735.975336] Key type ._llcrypt registered [ 3735.976298] Key type .llcrypt registered [ 3736.018124] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3736.440771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3736.723224] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 3736.806980] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 3736.808458] LNet: Accept secure, port 988 [ 3738.423113] Key type lgssc registered [ 3738.900549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3742.270334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3743.467436] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3743.491939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3745.672808] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3748.832452] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3749.606955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3749.780815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3751.518578] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3754.990734] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3757.207341] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3759.915347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3760.024390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3761.446893] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3764.781565] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3773.350217] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 3780.577208] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3780.582200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3780.590187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3781.089281] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3781.092267] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3781.096820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3783.649117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3783.651067] Lustre: Skipped 1 previous similar message [ 3788.768900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3788.771793] Lustre: Skipped 1 previous similar message [ 3793.887225] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3793.889745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3793.896051] Lustre: Skipped 1 previous similar message [ 3793.966602] Lustre: server umount lustre-OST0000 complete [ 3795.936502] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3795.942052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3799.010206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3801.759155] Lustre: server umount lustre-MDT0000 complete [ 3803.362988] LustreError: 285975:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771972324 with bad export cookie 11325174791845559466 [ 3803.365067] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3803.371241] LustreError: 285975:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3803.479190] Lustre: server umount lustre-MDT0001 complete [ 3808.447155] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 3809.369603] Key type lgssc unregistered [ 3809.472287] LNet: 289836:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3809.474764] LNet: Removed LNI 192.168.204.111@tcp [ 3809.769754] Key type .llcrypt unregistered [ 3809.770959] Key type ._llcrypt unregistered [ 3817.328963] alg: No test for adler32 (adler32-zlib) [ 3818.081316] Key type ._llcrypt registered [ 3818.082363] Key type .llcrypt registered [ 3818.123668] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3818.433317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3818.642065] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 3818.705920] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 3818.707255] LNet: Accept secure, port 988 [ 3820.279068] Key type lgssc registered [ 3820.633513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3824.152404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3825.375313] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3825.396197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3827.448109] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3830.753414] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3831.596592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3831.791523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3833.752742] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3836.958673] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3839.269075] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3841.953499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3842.056779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3843.350462] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3846.295907] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3855.403643] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 3860.452585] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3860.453963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3860.459227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3860.465989] Lustre: Skipped 1 previous similar message [ 3861.800083] Lustre: server umount lustre-OST0000 complete [ 3865.570602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3865.579550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3865.583442] Lustre: Skipped 2 previous similar messages [ 3869.651022] Lustre: server umount lustre-MDT0000 complete [ 3870.688487] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3870.948548] LustreError: 293677:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771972391 with bad export cookie 15755528263705170751 [ 3870.953391] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3870.954137] LustreError: 293677:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3871.103075] Lustre: server umount lustre-MDT0001 complete [ 3876.119618] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 3877.273797] Key type lgssc unregistered [ 3877.400383] LNet: 294780:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3877.403684] LNet: Removed LNI 192.168.204.111@tcp [ 3877.789069] Key type .llcrypt unregistered [ 3877.790835] Key type ._llcrypt unregistered [ 3887.873843] alg: No test for adler32 (adler32-zlib) [ 3888.626303] Key type ._llcrypt registered [ 3888.627173] Key type .llcrypt registered [ 3888.663483] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3889.032073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3889.179482] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 3889.245259] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 3889.247974] LNet: Accept secure, port 988 [ 3890.823168] Key type lgssc registered [ 3891.314734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3894.548114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3895.710568] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3895.727893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3897.667809] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3900.898731] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3901.742795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3901.908736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3903.663316] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3907.174837] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3909.797518] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3912.948115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3913.087695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3914.952920] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3918.079516] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3925.416333] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 3937.760829] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3937.767081] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3937.774137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3940.833661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3940.834081] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3940.838622] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3943.200983] Lustre: server umount lustre-OST0000 complete [ 3945.953723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3945.961419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3945.964525] Lustre: Skipped 2 previous similar messages [ 3950.872581] Lustre: server umount lustre-MDT0000 complete [ 3951.072751] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3951.080885] LustreError: Skipped 1 previous similar message [ 3952.698450] LustreError: 296220:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771972473 with bad export cookie 2524783001498595336 [ 3952.702171] LustreError: 296220:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3952.703171] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3952.861761] Lustre: server umount lustre-MDT0001 complete [ 3958.224454] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 3959.647812] Key type lgssc unregistered [ 3959.825467] LNet: 300732:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3959.830227] LNet: Removed LNI 192.168.204.111@tcp [ 3960.307539] Key type .llcrypt unregistered [ 3960.309553] Key type ._llcrypt unregistered [ 3968.193337] alg: No test for adler32 (adler32-zlib) [ 3968.945463] Key type ._llcrypt registered [ 3968.946528] Key type .llcrypt registered [ 3968.991286] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3969.492977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3969.675212] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 3969.748557] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 3969.750237] LNet: Accept secure, port 988 [ 3971.335207] Key type lgssc registered [ 3971.917744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3975.374201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3976.597192] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3976.615454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3978.767004] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3981.793793] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3983.005292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3983.184375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3985.137824] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3988.768545] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3991.495586] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3994.157277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3994.265230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3995.596883] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 3998.198102] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4006.570269] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 4015.074684] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4015.080396] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4015.087617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4017.125739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4017.126635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4019.418965] Lustre: server umount lustre-OST0000 complete [ 4022.240284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4022.241069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4022.243609] Lustre: Skipped 1 previous similar message [ 4022.245594] Lustre: Skipped 1 previous similar message [ 4027.143937] Lustre: server umount lustre-MDT0000 complete [ 4027.361512] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4027.368408] LustreError: Skipped 1 previous similar message [ 4028.604988] LustreError: 301977:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771972549 with bad export cookie 16644825373714661551 [ 4028.606930] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4028.607895] LustreError: 301977:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4028.737329] Lustre: server umount lustre-MDT0001 complete [ 4034.159024] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 4035.415856] Key type lgssc unregistered [ 4035.543420] LNet: 305854:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4035.548069] LNet: Removed LNI 192.168.204.111@tcp [ 4035.928886] Key type .llcrypt unregistered [ 4035.930156] Key type ._llcrypt unregistered [ 4044.184119] alg: No test for adler32 (adler32-zlib) [ 4044.936371] Key type ._llcrypt registered [ 4044.937938] Key type .llcrypt registered [ 4044.972927] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4045.274769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4045.480868] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 4045.547374] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 4045.549031] LNet: Accept secure, port 988 [ 4047.119143] Key type lgssc registered [ 4047.530971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4050.939342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4052.103996] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4052.119444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4053.896664] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4057.251316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4057.371674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4058.838357] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4062.165704] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4065.072424] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4068.141646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4068.245056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4069.922302] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4073.115678] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4082.600974] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:257 [ 4087.778922] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4087.780913] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4087.782637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4087.784624] LustreError: Skipped 1 previous similar message [ 4087.791867] Lustre: Skipped 1 previous similar message [ 4088.821512] Lustre: server umount lustre-OST0000 complete [ 4092.897299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4092.898966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4092.905557] Lustre: Skipped 1 previous similar message [ 4092.911804] Lustre: Skipped 2 previous similar messages [ 4096.732093] Lustre: server umount lustre-MDT0000 complete [ 4098.017180] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4098.027126] LustreError: Skipped 1 previous similar message [ 4098.455738] LustreError: 307101:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771972619 with bad export cookie 7765981354012865206 [ 4098.460514] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4098.464418] LustreError: 307101:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4098.605329] Lustre: server umount lustre-MDT0001 complete [ 4104.296801] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 4105.676671] Key type lgssc unregistered [ 4105.811410] LNet: 310792:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4105.815928] LNet: Removed LNI 192.168.204.111@tcp [ 4106.231985] Key type .llcrypt unregistered [ 4106.233772] Key type ._llcrypt unregistered [ 4111.657808] alg: No test for adler32 (adler32-zlib) [ 4112.410316] Key type ._llcrypt registered [ 4112.411134] Key type .llcrypt registered [ 4112.469700] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4120.641530] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 4127.707985] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4128.057398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4128.312822] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 4128.388942] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 4128.390375] LNet: Accept secure, port 988 [ 4129.975120] Key type lgssc registered [ 4130.518872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4134.023826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4136.982903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4139.706346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4142.592289] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4145.730684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4145.767791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4146.896617] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4146.916674] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4146.970166] Lustre: lustre-MDT0000: new disk, initializing [ 4147.007422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4147.015853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4148.990497] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4154.175997] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4154.222767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4154.273091] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4154.308319] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4154.310185] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4154.311702] Lustre: Skipped 1 previous similar message [ 4154.355379] Lustre: lustre-MDT0001: new disk, initializing [ 4154.393846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4154.411606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4154.417664] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4156.264054] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4160.856776] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4163.639968] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4166.842746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4166.876119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4166.879992] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4167.005458] Lustre: lustre-OST0000: new disk, initializing [ 4167.008706] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4167.044420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4168.714346] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4171.764032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4171.766513] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4172.869060] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4174.691969] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4174.787628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4176.624801] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4176.684152] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4181.984380] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4181.985889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4181.987155] LustreError: Skipped 1 previous similar message [ 4181.987270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4181.992964] Lustre: Skipped 1 previous similar message [ 4183.538278] Lustre: server umount lustre-OST0000 complete [ 4187.105029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4187.106028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4187.110716] Lustre: Skipped 1 previous similar message [ 4187.114135] Lustre: Skipped 1 previous similar message [ 4192.224298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4192.228940] Lustre: Skipped 1 previous similar message [ 4197.344097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4197.346701] Lustre: Skipped 1 previous similar message [ 4199.391143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4199.464634] Lustre: server umount lustre-MDT0000 complete [ 4201.076846] LustreError: 314529:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771972721 with bad export cookie 6866413759423131571 [ 4201.081985] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4201.215903] Lustre: server umount lustre-MDT0001 complete [ 4204.300425] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 17:38:44 (1771972724) [ 4207.442665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4207.646681] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4207.666327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4209.496969] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4212.706471] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4213.225159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4213.421611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4215.258483] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4218.885735] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4221.555623] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4224.725335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4224.883321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4226.543635] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4229.686488] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4244.449057] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4244.454382] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4244.462613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4248.548063] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4248.548927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4249.590297] Lustre: server umount lustre-OST0000 complete [ 4253.664688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4257.415837] Lustre: server umount lustre-MDT0000 complete [ 4258.784673] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4258.791875] LustreError: Skipped 1 previous similar message [ 4258.968744] LustreError: 319071:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771972779 with bad export cookie 6866413759423132425 [ 4258.971898] LustreError: 319881:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4258.974540] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4258.977772] LustreError: 319071:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4258.984755] LustreError: 319881:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 llog-records: rc = -5 [ 4259.001682] LustreError: 319881:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -5 [ 4264.729368] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 4266.110646] Key type lgssc unregistered [ 4266.236443] LNet: 322768:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4266.241321] LNet: Removed LNI 192.168.204.111@tcp [ 4266.707575] Key type .llcrypt unregistered [ 4266.711226] Key type ._llcrypt unregistered [ 4279.656839] alg: No test for adler32 (adler32-zlib) [ 4280.409432] Key type ._llcrypt registered [ 4280.410349] Key type .llcrypt registered [ 4280.451840] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 4287.132712] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4287.559921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4287.822721] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 4287.896246] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 4287.898462] LNet: Accept secure, port 988 [ 4289.487137] Key type lgssc registered [ 4290.060389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4294.311128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4297.395036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4300.498358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4303.385891] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4305.768783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4308.043540] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4311.071901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4311.112561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4312.238288] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4312.252726] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4312.315699] Lustre: lustre-MDT0000: new disk, initializing [ 4312.344260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4312.351968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4314.471124] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4319.994052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4320.036713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4320.090216] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4320.122159] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4320.125634] Lustre: Skipped 1 previous similar message [ 4320.177850] Lustre: lustre-MDT0001: new disk, initializing [ 4320.232730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4320.255079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4320.261398] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4322.413850] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4327.599895] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4330.490443] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4333.347345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4333.373052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4333.376814] Lustre: lustre-OST2710-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4333.487279] Lustre: lustre-OST2710: new disk, initializing [ 4333.488865] Lustre: srv-lustre-OST2710: No data found on store. Initialize space: rc = -61 [ 4333.516199] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4335.450218] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4335.573362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4335.581885] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4340.259828] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4343.149547] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4343.184290] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4343.188244] Lustre: lustre-OST03e8-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4343.249791] Lustre: lustre-OST03e8: new disk, initializing [ 4343.252142] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space: rc = -61 [ 4343.300081] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4344.983575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4344.989773] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4345.178758] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4348.292287] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4351.708566] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 40 [ 4351.863628] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4353.661989] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 40 [ 4353.759520] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4355.470967] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 40 [ 4355.580362] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4357.252613] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 40 [ 4357.343206] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4360.678084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4360.682092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4360.686016] Lustre: Skipped 2 previous similar messages [ 4360.693090] Lustre: Skipped 3 previous similar messages [ 4364.775379] Lustre: server umount lustre-MDT0000 complete [ 4365.793659] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4365.801232] LustreError: Skipped 3 previous similar messages [ 4366.492883] LustreError: 327560:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771972887 with bad export cookie 10096271197264321299 [ 4366.499226] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4366.500117] LustreError: 327560:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4366.679342] Lustre: server umount lustre-MDT0001 complete [ 4375.007225] Lustre: 332193:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771972889/real 1771972889] req@000000009aefbe9b x1858048159146624/t0(0) o39->lustre-MDT0001-lwp-OST2710@0@lo:12/10 lens 224/224 e 0 to 1 dl 1771972895 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4375.050936] Lustre: server umount lustre-OST2710 complete [ 4383.199183] Lustre: 332394:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771972897/real 1771972897] req@000000003f273a61 x1858048159147136/t0(0) o39->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1771972903 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4383.332565] Lustre: server umount lustre-OST03e8 complete [ 4391.959730] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 4396.261748] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4400.572866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4403.837720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4406.211677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4408.641212] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4411.483794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4411.524863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4411.640337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4411.653951] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4411.690781] Lustre: lustre-MDT0000: new disk, initializing [ 4411.721701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4411.729878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4413.471702] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4417.857648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4417.890269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4417.933300] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4418.044979] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4419.775689] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4424.106941] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4426.851118] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4429.959238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4429.996918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4430.001561] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4430.144141] Lustre: lustre-OST0000: new disk, initializing [ 4430.147019] Lustre: Skipped 1 previous similar message [ 4430.150473] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4430.154515] Lustre: Skipped 2 previous similar messages [ 4430.194731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4430.200609] Lustre: Skipped 1 previous similar message [ 4432.020620] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4436.841044] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4437.509543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4437.513317] Lustre: Skipped 1 previous similar message [ 4437.515854] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4438.734736] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4438.853371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4440.390017] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4440.488137] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4442.595931] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4442.596789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4442.598096] LustreError: Skipped 1 previous similar message [ 4442.601311] Lustre: Skipped 1 previous similar message [ 4442.603300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4442.606624] Lustre: Skipped 1 previous similar message [ 4443.445636] Lustre: server umount lustre-OST0000 complete [ 4447.713682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4447.715278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4447.717685] Lustre: Skipped 1 previous similar message [ 4447.720440] Lustre: Skipped 1 previous similar message [ 4452.834094] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4452.843309] LustreError: Skipped 1 previous similar message [ 4452.997665] LustreError: 335293:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771972973 with bad export cookie 10096271197264322916 [ 4453.003621] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4453.005301] LustreError: 335293:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4453.155397] Lustre: server umount lustre-MDT0001 complete [ 4453.156594] Lustre: Skipped 1 previous similar message [ 4457.198639] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 17:42:57 (1771972977) [ 4457.985484] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4458.744812] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 17:42:58 (1771972978) [ 4460.216632] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4463.489520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4465.216749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4467.040353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4468.965130] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4469.544521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4472.767456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4472.903783] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4472.916089] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4472.991714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4475.099564] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4479.407161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4479.453984] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4481.603502] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4485.084594] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4487.802873] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4491.089761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4491.134143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4491.266090] Lustre: Denying initial registration attempt from nid 192.168.204.111@tcp, specified as failover [ 4491.271970] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4491.275309] LustreError: 343322:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -99 [ 4491.278177] LustreError: 343322:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 4491.280720] LustreError: 343322:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 4491.362551] Lustre: server umount lustre-OST0000 complete [ 4491.365095] LustreError: 343322:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4494.816495] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4494.822603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4494.832401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4501.475229] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4501.484597] LustreError: Skipped 1 previous similar message [ 4501.655239] LustreError: 342166:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771973022 with bad export cookie 10096271197264323756 [ 4501.658353] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4501.662453] LustreError: 342166:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4507.599628] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 4509.105796] Key type lgssc unregistered [ 4509.266602] LNet: 344394:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4509.271580] LNet: Removed LNI 192.168.204.111@tcp [ 4509.715911] Key type .llcrypt unregistered [ 4509.718241] Key type ._llcrypt unregistered [ 4515.683286] alg: No test for adler32 (adler32-zlib) [ 4516.435395] Key type ._llcrypt registered [ 4516.436313] Key type .llcrypt registered [ 4516.471209] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4516.799438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4517.065917] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 4517.138211] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 4517.139782] LNet: Accept secure, port 988 [ 4518.727160] Key type lgssc registered [ 4519.313191] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4522.705396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4524.543374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4526.393847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4528.100207] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4528.615847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4535.614680] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4539.619078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4540.758006] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4540.769982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4540.859900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4542.870247] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4547.008821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4547.052361] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4547.067941] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4547.188716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4549.256898] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4553.109923] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4555.987406] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4559.324390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4559.369618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4559.481605] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4559.585596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4561.755784] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4565.573642] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4570.082268] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4570.083210] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4570.087870] LustreError: Skipped 1 previous similar message [ 4570.097138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4570.103072] Lustre: Skipped 1 previous similar message [ 4572.737455] Lustre: server umount lustre-OST0000 complete [ 4575.201453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4575.202563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4575.209355] Lustre: Skipped 2 previous similar messages [ 4580.320658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4580.324663] Lustre: Skipped 2 previous similar messages [ 4580.762154] Lustre: server umount lustre-MDT0000 complete [ 4582.542549] LustreError: 348065:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771973103 with bad export cookie 3800498533220157199 [ 4582.546677] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4582.549517] LustreError: 348065:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4582.698489] Lustre: server umount lustre-MDT0001 complete [ 4588.157320] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 4589.728945] Key type lgssc unregistered [ 4589.899428] LNet: 350961:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4589.905330] LNet: Removed LNI 192.168.204.111@tcp [ 4590.357343] Key type .llcrypt unregistered [ 4590.359843] Key type ._llcrypt unregistered [ 4602.460562] alg: No test for adler32 (adler32-zlib) [ 4603.213367] Key type ._llcrypt registered [ 4603.214327] Key type .llcrypt registered [ 4603.251962] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4603.584220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4603.870187] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 4603.948809] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 4603.950168] LNet: Accept secure, port 988 [ 4605.535095] Key type lgssc registered [ 4606.145748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4610.147965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4611.386738] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4611.414233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4613.576795] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4616.673816] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4617.706659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4617.898736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4620.141421] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4622.819666] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4625.035464] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4627.829386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4628.027169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4630.151468] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4633.931297] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4647.274815] LustreError: 355088:0:(ldlm_resource.c:1127:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200001b71:0xf:0x0].0x0 (000000003a19d51b) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4648.928502] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4648.933741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4648.941898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4649.470467] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.11@tcp (stopping) [ 4651.489795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4651.492414] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4653.479441] Lustre: server umount lustre-MDT0000 complete [ 4654.576696] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4655.320382] LustreError: 352402:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771973176 with bad export cookie 15206525335937588324 [ 4655.326786] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4656.607888] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4656.608072] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4656.615800] Lustre: Skipped 1 previous similar message [ 4656.617887] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4656.619469] Lustre: Skipped 2 previous similar messages [ 4660.640486] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4660.640655] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4660.646439] LustreError: Skipped 1 previous similar message [ 4660.652884] Lustre: Skipped 1 previous similar message [ 4661.508786] Lustre: server umount lustre-MDT0001 complete [ 4663.787195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4667.377973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4672.992522] Lustre: Evicted from MGS (at 192.168.204.111@tcp) after server handle changed from 0xd3086e39b4899c64 to 0xd3086e39b489b841 [ 4672.995592] Lustre: MGC192.168.204.111@tcp: Connection restored to 192.168.204.111@tcp (at 0@lo) [ 4673.007609] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4673.113190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4673.136549] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:193 [ 4674.330429] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4676.780293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4677.926091] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 4678.148110] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4680.352840] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4683.043549] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 4683.911313] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4685.552198] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4686.067060] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4687.327970] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4687.329938] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4687.333137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4688.353718] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4693.336723] Lustre: server umount lustre-OST0000 complete [ 4697.567682] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4697.569745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4697.572505] Lustre: Skipped 1 previous similar message [ 4700.546406] Lustre: server umount lustre-MDT0000 complete [ 4701.658967] LustreError: 358131:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771973222 with bad export cookie 15206525335937595457 [ 4701.661638] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4701.663065] LustreError: 358131:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4706.172598] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 4706.955306] Key type lgssc unregistered [ 4707.043294] LNet: 358933:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4707.045672] LNet: Removed LNI 192.168.204.111@tcp [ 4707.276232] Key type .llcrypt unregistered [ 4707.277161] Key type ._llcrypt unregistered [ 4715.078824] alg: No test for adler32 (adler32-zlib) [ 4715.830278] Key type ._llcrypt registered [ 4715.831191] Key type .llcrypt registered [ 4715.861245] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4716.157147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4716.344237] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 4716.408469] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 4716.409775] LNet: Accept secure, port 988 [ 4717.983074] Key type lgssc registered [ 4718.259539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4720.479382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4721.611843] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4721.625640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4722.856631] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4725.466643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4725.576244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4727.539402] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4731.008078] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4733.790332] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4735.969698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4735.970036] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4735.970849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4735.977063] Lustre: Skipped 1 previous similar message [ 4741.090077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4741.094105] Lustre: Skipped 2 previous similar messages [ 4741.414540] Lustre: server umount lustre-MDT0000 complete [ 4743.250088] LustreError: 360371:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771973263 with bad export cookie 2660898041143595801 [ 4743.256716] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4743.256920] LustreError: 360371:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4743.448694] Lustre: server umount lustre-MDT0001 complete [ 4750.401477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4750.560137] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4750.576786] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4750.689250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4752.876932] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4757.004251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4757.052455] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4757.069418] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4757.188687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4758.806773] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4762.331544] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4765.121466] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4768.524545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4768.626791] LustreError: 13b-9: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4768.633073] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4768.637735] LustreError: 365085:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 4768.641118] LustreError: 365085:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 4768.644158] LustreError: 365085:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 4768.702200] Lustre: server umount lustre-OST0000 complete [ 4768.704353] LustreError: 365085:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4771.829220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4771.941683] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4772.036658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4774.016483] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4777.518383] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:225 [ 4777.630169] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4781.307886] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4781.344912] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4781.350672] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4781.395680] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4781.443914] Lustre: lustre-OST0001: new disk, initializing [ 4781.447391] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 4781.500171] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4782.681516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4782.685142] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4783.594591] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4787.005566] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4793.319424] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4793.319742] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4793.324571] LustreError: Skipped 1 previous similar message [ 4793.325442] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4793.331956] Lustre: Skipped 1 previous similar message [ 4794.545404] Lustre: server umount lustre-OST0001 complete [ 4797.920292] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4797.920772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4797.924770] LustreError: Skipped 1 previous similar message [ 4797.925500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4797.931698] Lustre: Skipped 1 previous similar message [ 4797.939640] Lustre: Skipped 2 previous similar messages [ 4798.436871] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4798.445757] LustreError: Skipped 1 previous similar message [ 4802.236856] Lustre: server umount lustre-OST0000 complete [ 4808.674545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4808.674577] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4808.675584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4808.675593] Lustre: Skipped 2 previous similar messages [ 4808.678951] Lustre: Skipped 1 previous similar message [ 4811.840703] LustreError: 363083:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771973332 with bad export cookie 2660898041143596340 [ 4811.846807] LustreError: 363083:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4811.847640] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4812.029831] Lustre: server umount lustre-MDT0001 complete [ 4812.032439] Lustre: Skipped 1 previous similar message [ 4816.888281] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 4818.461723] Key type lgssc unregistered [ 4818.627686] LNet: 368451:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4818.633086] LNet: Removed LNI 192.168.204.111@tcp [ 4819.109204] Key type .llcrypt unregistered [ 4819.111788] Key type ._llcrypt unregistered [ 4835.424564] alg: No test for adler32 (adler32-zlib) [ 4836.176302] Key type ._llcrypt registered [ 4836.177109] Key type .llcrypt registered [ 4836.229222] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 4843.360082] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4843.909432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4844.058155] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 4844.130474] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 4844.131996] LNet: Accept secure, port 988 [ 4845.711056] Key type lgssc registered [ 4846.169588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4850.582760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4853.865853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4856.950154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4860.246774] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4863.592374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4863.635490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4864.780889] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4864.799502] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4864.850010] Lustre: lustre-MDT0000: new disk, initializing [ 4864.892673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4864.908255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4867.156601] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4872.843695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4872.880070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4872.936773] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4872.969793] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4872.973314] Lustre: Skipped 1 previous similar message [ 4873.019258] Lustre: lustre-MDT0001: new disk, initializing [ 4873.054315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4873.071620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4873.075904] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4875.169422] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4880.362161] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4883.289422] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4886.617689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4886.658119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4886.664105] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4886.809391] Lustre: lustre-OST0000: new disk, initializing [ 4886.812601] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4886.853872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4888.967632] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4892.179692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4892.189236] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4893.984733] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4896.247893] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4896.386224] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4898.619748] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4898.763226] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4902.372191] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4902.375927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4902.377329] LustreError: Skipped 1 previous similar message [ 4902.386384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4902.391951] Lustre: Skipped 1 previous similar message [ 4907.492837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4907.496913] Lustre: Skipped 1 previous similar message [ 4912.612071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4912.616099] Lustre: Skipped 1 previous similar message [ 4914.143177] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4914.249527] Lustre: server umount lustre-OST0000 complete [ 4917.730292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4917.731290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4917.737691] Lustre: Skipped 2 previous similar messages [ 4917.747760] Lustre: Skipped 1 previous similar message [ 4922.273539] Lustre: server umount lustre-MDT0000 complete [ 4922.848741] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4922.854023] LustreError: Skipped 1 previous similar message [ 4923.967916] LustreError: 374628:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771973444 with bad export cookie 14659950190231127443 [ 4923.972705] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4923.973716] LustreError: 374628:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4924.111299] Lustre: server umount lustre-MDT0001 complete [ 4927.437207] Lustre: DEBUG MARKER: == conf-sanity test 60: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 17:50:47 (1771973447) [ 4930.322994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4932.612269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4942.943849] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 4947.152126] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4951.516339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4954.640326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4957.400433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4960.528179] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4963.599233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4963.630485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4963.730985] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4963.743907] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4963.792360] Lustre: lustre-MDT0000: new disk, initializing [ 4963.817979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4963.831302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4965.279846] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4970.527548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4970.590665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4970.650954] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4970.693740] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4970.695593] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4970.698390] Lustre: Skipped 1 previous similar message [ 4970.746828] Lustre: lustre-MDT0001: new disk, initializing [ 4970.781066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4970.797962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4970.802594] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4972.587542] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4977.447362] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4980.193748] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4983.462928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4983.501972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4983.507413] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4983.644711] Lustre: lustre-OST0000: new disk, initializing [ 4983.646921] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4983.681196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4985.449784] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 4990.535299] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4990.982159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4990.988178] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4992.641787] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4992.774312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4994.860199] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4994.978818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4996.065851] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4996.067918] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4996.072038] LustreError: Skipped 1 previous similar message [ 4996.083921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4996.090505] Lustre: Skipped 1 previous similar message [ 4998.008353] Lustre: server umount lustre-OST0000 complete [ 5001.185545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5001.196650] Lustre: Skipped 2 previous similar messages [ 5006.047341] Lustre: server umount lustre-MDT0000 complete [ 5006.303871] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5006.308288] LustreError: Skipped 1 previous similar message [ 5007.857189] LustreError: 380294:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771973528 with bad export cookie 14659950190231128297 [ 5007.860136] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5007.862515] LustreError: 380294:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5011.370088] Lustre: DEBUG MARKER: == conf-sanity test 61: large xattr ====================== 17:52:11 (1771973531) [ 5014.836294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5015.045923] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5015.076750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5017.079806] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5020.130107] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5021.057973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5023.238862] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5026.422463] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5028.995170] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5032.264361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5034.415572] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5037.832435] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5052.275278] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61.conf-sanity [ 5053.123801] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61.conf-sanity [ 5054.008994] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61.conf-sanity [ 5054.859472] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61.conf-sanity after remounting MDS [ 5055.778345] Lustre: Failing over lustre-MDT0000 [ 5056.007236] Lustre: server umount lustre-MDT0000 complete [ 5056.008384] Lustre: Skipped 1 previous similar message [ 5056.992958] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5056.996695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5057.004931] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5065.184959] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5065.198548] LustreError: Skipped 7 previous similar messages [ 5067.167152] Lustre: 370766:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771973580/real 1771973580] req@0000000027aebe43 x1858048742174080/t0(0) o400->MGC192.168.204.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1771973587 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5067.179114] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5070.270977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5072.357145] Lustre: Evicted from MGS (at 192.168.204.111@tcp) after server handle changed from 0xcb729b06b9a58c3f to 0xcb729b06b9a59021 [ 5072.363885] Lustre: MGC192.168.204.111@tcp: Connection restored to 192.168.204.111@tcp (at 0@lo) [ 5072.480654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5072.485983] Lustre: Skipped 2 previous similar messages [ 5072.527054] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5074.496649] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5076.976654] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5077.991096] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 5078.001796] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5078.032882] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 5080.691782] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5081.473055] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5083.640471] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61.conf-sanity [ 5084.537888] Lustre: Failing over lustre-MDT0000 [ 5084.709755] Lustre: server umount lustre-MDT0000 complete [ 5087.231113] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5087.239543] LustreError: Skipped 9 previous similar messages [ 5088.226161] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5088.226222] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5088.238450] Lustre: Skipped 4 previous similar messages [ 5094.367171] Lustre: 370765:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771973608/real 1771973608] req@000000005633b3b2 x1858048742181824/t0(0) o400->MGC192.168.204.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1771973615 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5094.379516] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5098.949910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5100.517853] Lustre: Evicted from MGS (at 192.168.204.111@tcp) after server handle changed from 0xcb729b06b9a59021 to 0xcb729b06b9a59323 [ 5100.528785] Lustre: MGC192.168.204.111@tcp: Connection restored to (at 0@lo) [ 5100.532509] Lustre: Skipped 2 previous similar messages [ 5100.691123] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5102.764561] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5104.624906] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5106.158303] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 5106.169658] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5106.199754] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [ 5109.049871] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5109.902604] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5116.386771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5116.387454] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5116.388471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5116.388477] Lustre: Skipped 2 previous similar messages [ 5116.395302] Lustre: Skipped 1 previous similar message [ 5118.814192] Lustre: server umount lustre-MDT0000 complete [ 5120.557758] LustreError: 384934:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771973641 with bad export cookie 14659950190231130915 [ 5120.562391] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5120.569112] LustreError: 384934:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5128.671165] Lustre: 390756:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771973643/real 1771973643] req@00000000bd47435a x1858048742191360/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1771973649 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5135.702068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5135.953675] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5135.964065] LustreError: Skipped 19 previous similar messages [ 5138.202374] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5142.443308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5142.585287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5142.588328] Lustre: Skipped 2 previous similar messages [ 5144.751829] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5148.478454] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5151.330502] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5154.525841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5156.172884] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5159.639980] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5166.953806] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [ 5172.194652] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5172.196707] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5172.197807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5172.197818] Lustre: Skipped 2 previous similar messages [ 5172.212930] Lustre: Skipped 2 previous similar messages [ 5190.111162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5190.177345] Lustre: server umount lustre-MDT0000 complete [ 5190.179354] Lustre: Skipped 3 previous similar messages [ 5191.919781] LustreError: 391563:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771973712 with bad export cookie 14659950190231131650 [ 5191.924995] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5191.930160] LustreError: 391563:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5197.738517] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 5199.168920] Key type lgssc unregistered [ 5199.308456] LNet: 395257:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5199.313807] LNet: Removed LNI 192.168.204.111@tcp [ 5199.738792] Key type .llcrypt unregistered [ 5199.740483] Key type ._llcrypt unregistered [ 5211.606720] alg: No test for adler32 (adler32-zlib) [ 5212.360266] Key type ._llcrypt registered [ 5212.361235] Key type .llcrypt registered [ 5212.398203] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 5212.702500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5212.920883] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 5212.990980] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 5212.992409] LNet: Accept secure, port 988 [ 5214.575118] Key type lgssc registered [ 5215.136348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5218.719822] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5218.724444] LustreError: 396740:0:(osd_handler.c:8215:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5218.728790] LustreError: 396740:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5218.732460] LustreError: 396740:0:(obd_mount.c:200:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5218.735923] LustreError: 396740:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5218.739752] LustreError: 396740:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5222.281314] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5222.286368] LustreError: 397137:0:(osd_handler.c:8215:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5222.292578] LustreError: 397137:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5222.297116] LustreError: 397137:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5222.301783] LustreError: 397137:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5222.307343] LustreError: 397137:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5230.573475] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 5231.813223] Key type lgssc unregistered [ 5231.966570] LNet: 398095:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5231.971619] LNet: Removed LNI 192.168.204.111@tcp [ 5232.386876] Key type .llcrypt unregistered [ 5232.389241] Key type ._llcrypt unregistered [ 5240.992358] alg: No test for adler32 (adler32-zlib) [ 5241.747614] Key type ._llcrypt registered [ 5241.748771] Key type .llcrypt registered [ 5241.812970] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 5249.724736] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 5250.095826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5250.368852] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 5250.440706] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 5250.442171] LNet: Accept secure, port 988 [ 5252.031116] Key type lgssc registered [ 5252.629414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5257.075366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5260.219464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5262.964366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5265.988426] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5268.395272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5268.414754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5269.488401] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5269.497109] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5269.531380] Lustre: lustre-MDT0000: new disk, initializing [ 5269.552451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5269.558205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5270.839863] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5274.557629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5274.581873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5274.609945] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5274.625860] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5274.627369] Lustre: Skipped 1 previous similar message [ 5274.660618] Lustre: lustre-MDT0001: new disk, initializing [ 5274.687455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5274.697521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5274.700354] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5275.906452] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5279.356211] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5281.013700] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5282.968607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5282.994393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5282.996447] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5283.075774] Lustre: lustre-OST0000: new disk, initializing [ 5283.077417] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5283.101479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5284.296855] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5287.670881] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5289.277886] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5292.040840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5292.047436] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5292.529750] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 5294.669630] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5294.801650] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5297.125809] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5297.127181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5297.131072] LustreError: Skipped 1 previous similar message [ 5297.140475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5297.145530] Lustre: Skipped 1 previous similar message [ 5301.886987] Lustre: server umount lustre-OST0000 complete [ 5305.312289] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5305.317927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5305.326352] Lustre: Skipped 1 previous similar message [ 5305.330131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5307.361381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5307.362415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5312.481160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5312.482679] Lustre: Skipped 2 previous similar messages [ 5317.601205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5317.605264] Lustre: Skipped 1 previous similar message [ 5318.111167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5318.184699] Lustre: server umount lustre-MDT0000 complete [ 5319.948997] LustreError: 401360:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771973840 with bad export cookie 12305998163003311541 [ 5319.955862] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5319.956348] LustreError: 401360:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5320.128538] Lustre: server umount lustre-MDT0001 complete [ 5324.273520] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5325.126359] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 17:57:25 (1771973845) [ 5328.125745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5328.388753] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5328.423992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5330.673564] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5333.474043] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5334.746031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5334.921841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5336.970954] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5340.729553] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5343.594227] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5346.837209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5347.026189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5349.053913] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5352.769705] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5356.030472] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5356.076138] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5356.081540] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 5356.163315] Lustre: lustre-OST0001: new disk, initializing [ 5356.167254] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5358.313336] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5359.887462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5359.894210] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5363.503914] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5369.857528] Lustre: lustre-OST0001: Not available for connect from 192.168.204.11@tcp (stopping) [ 5370.338237] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5370.344103] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5370.352694] Lustre: Skipped 1 previous similar message [ 5380.063138] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5380.082672] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5380.091808] Lustre: server umount lustre-OST0001 complete [ 5385.700079] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5385.704317] LustreError: Skipped 3 previous similar messages [ 5390.822073] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5390.830515] LustreError: Skipped 1 previous similar message [ 5395.424890] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5395.430135] LustreError: Skipped 1 previous similar message [ 5395.433021] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5395.441588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5395.445518] Lustre: Skipped 5 previous similar messages [ 5398.779248] Lustre: server umount lustre-OST0000 complete [ 5406.882760] Lustre: server umount lustre-MDT0000 complete [ 5408.783064] LustreError: 405965:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771973929 with bad export cookie 12305998163003312381 [ 5408.789375] LustreError: 405965:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5408.789414] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5423.071179] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5423.237766] Lustre: server umount lustre-MDT0001 complete [ 5429.043556] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 5430.544838] Key type lgssc unregistered [ 5430.710582] LNet: 410895:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5430.716095] LNet: Removed LNI 192.168.204.111@tcp [ 5431.194190] Key type .llcrypt unregistered [ 5431.196304] Key type ._llcrypt unregistered [ 5446.889761] alg: No test for adler32 (adler32-zlib) [ 5447.642278] Key type ._llcrypt registered [ 5447.643201] Key type .llcrypt registered [ 5447.683184] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 5455.229213] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 5455.649148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5455.799403] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 5455.890887] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 5455.892499] LNet: Accept secure, port 988 [ 5457.471100] Key type lgssc registered [ 5458.070466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5462.492563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5465.600686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5468.560492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5471.665067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5474.925343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5474.967325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5476.101832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5476.120156] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5476.176621] Lustre: lustre-MDT0000: new disk, initializing [ 5476.215681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5476.227218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5478.381082] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5483.917612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5483.955839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5484.009700] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5484.041443] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5484.045871] Lustre: Skipped 1 previous similar message [ 5484.102195] Lustre: lustre-MDT0001: new disk, initializing [ 5484.154118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5484.175217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5484.181494] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5486.200177] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5491.259167] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5493.970845] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5497.134395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5497.175412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5497.180777] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5497.313506] Lustre: lustre-OST0000: new disk, initializing [ 5497.316885] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5497.358802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5499.420131] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5501.127690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5501.133684] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5504.438513] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5506.648098] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5506.783683] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5508.542583] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5508.641970] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5511.649888] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5511.651777] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5511.654718] LustreError: Skipped 1 previous similar message [ 5511.663816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5511.669410] Lustre: Skipped 1 previous similar message [ 5515.637176] Lustre: server umount lustre-OST0000 complete [ 5520.352385] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5520.357346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5520.364452] Lustre: Skipped 1 previous similar message [ 5520.367515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5521.888551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5521.889335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5523.547261] Lustre: server umount lustre-MDT0000 complete [ 5525.268557] LustreError: 414978:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771974045 with bad export cookie 16952587018796877253 [ 5525.272382] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5525.275597] LustreError: 414978:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5525.412428] Lustre: server umount lustre-MDT0001 complete [ 5529.223597] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 18:00:49 (1771974049) [ 5533.470604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5537.356449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5537.562743] Lustre: lustre-MDT0000: new disk, initializing [ 5537.575747] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5537.606361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5539.580507] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5542.882388] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5543.548451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5543.705720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5545.615351] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5549.100932] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5551.647713] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5554.145062] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5554.145988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5554.158438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5554.161977] Lustre: Skipped 2 previous similar messages [ 5558.743334] Lustre: server umount lustre-MDT0000 complete [ 5559.264919] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5559.270646] LustreError: Skipped 1 previous similar message [ 5560.471844] LustreError: 420927:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771974081 with bad export cookie 16952587018796878107 [ 5560.476116] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5560.478810] LustreError: 420927:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5564.925882] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 18:01:24 (1771974084) [ 5567.938812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5568.148101] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5568.174767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5570.124752] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5573.601700] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5574.038774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5576.122234] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5579.600686] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5582.212806] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5585.321946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5585.470778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5585.473160] Lustre: Skipped 1 previous similar message [ 5587.375251] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5590.845810] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5604.655866] Lustre: Permanently deactivating lustre-OST0000 [ 5604.664790] Lustre: Setting parameter lustre-OST0000-osc.osc.active in log lustre-client [ 5613.563816] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5615.328260] LustreError: 425948:0:(mgs_llog.c:1551:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002020 [ 5615.334857] LustreError: 425948:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 5615.339311] LustreError: 425948:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 5616.132822] LustreError: 425996:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 5616.137461] LustreError: 425996:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 5623.158271] Lustre: server umount lustre-OST0000 complete [ 5623.160721] Lustre: Skipped 1 previous similar message [ 5624.446669] LustreError: 426245:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 5624.451312] LustreError: 426245:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 5629.921320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5629.922322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5629.925544] Lustre: Skipped 2 previous similar messages [ 5631.455159] Lustre: 426343:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771974146/real 1771974146] req@0000000075659d29 x1858049383897216/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1771974152 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5631.570315] Lustre: server umount lustre-MDT0000 complete [ 5633.277540] LustreError: 424689:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771974154 with bad export cookie 16952587018796878660 [ 5633.283057] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5633.284666] LustreError: 424689:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5639.647206] Lustre: 426546:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771974154/real 1771974154] req@000000006acc322c x1858049383898048/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1771974160 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5643.813975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5645.841317] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5647.109469] LustreError: 427454:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -6 [ 5650.027681] Lustre: 427799:0:(mgs_llog.c:1311:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.111@tcp [ 5657.055140] Lustre: 427898:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771974171/real 1771974171] req@0000000075659d29 x1858049383898496/t0(0) o251->MGC192.168.204.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1771974177 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5657.076807] Lustre: server umount MGS complete [ 5657.078952] Lustre: Skipped 1 previous similar message [ 5662.317352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5662.517489] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5662.521974] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5662.524466] Lustre: Skipped 1 previous similar message [ 5662.552213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5664.539768] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5668.501109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5668.628383] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5670.587828] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5674.125445] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5676.780486] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5677.670904] Lustre: Permanently reactivating lustre-OST0000 [ 5677.679634] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 5677.683681] Lustre: Skipped 2 previous similar messages [ 5692.384254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5694.472970] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5697.819332] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5713.890048] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5713.895051] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5713.902954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5713.906628] Lustre: Skipped 1 previous similar message [ 5724.127151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5724.219351] Lustre: server umount lustre-OST0000 complete [ 5733.867638] LustreError: 428509:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771974254 with bad export cookie 16952587018796879605 [ 5733.872819] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5733.874685] LustreError: 428509:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5739.730296] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 5741.146979] Key type lgssc unregistered [ 5741.301440] LNet: 432744:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5741.306643] LNet: Removed LNI 192.168.204.111@tcp [ 5741.738384] Key type .llcrypt unregistered [ 5741.740551] Key type ._llcrypt unregistered [ 5749.988550] alg: No test for adler32 (adler32-zlib) [ 5750.741237] Key type ._llcrypt registered [ 5750.742000] Key type .llcrypt registered [ 5750.777332] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 5758.336555] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 5758.727569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5758.911354] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 5758.991718] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 5758.993611] LNet: Accept secure, port 988 [ 5760.591133] Key type lgssc registered [ 5761.156360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5765.538664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5768.745900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5771.713355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5774.903080] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5778.241894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5778.284311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5779.405894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5779.422301] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5779.479561] Lustre: lustre-MDT0000: new disk, initializing [ 5779.550507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5779.563112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5781.660127] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5787.246662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5787.294621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5787.341720] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5787.368915] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5787.373276] Lustre: Skipped 1 previous similar message [ 5787.433553] Lustre: lustre-MDT0001: new disk, initializing [ 5787.482717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5787.502727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5787.508592] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5789.669500] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5794.792423] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5797.468547] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5800.601813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5800.642367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5800.647554] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5800.778604] Lustre: lustre-OST0000: new disk, initializing [ 5800.781754] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5800.821282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5802.750787] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5807.108558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5807.114514] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5807.605593] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5809.478814] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5809.618560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5811.726701] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5811.870219] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5817.314925] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5817.316308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5817.319590] LustreError: Skipped 1 previous similar message [ 5817.327468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5817.332615] Lustre: Skipped 1 previous similar message [ 5822.436885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5822.440527] Lustre: Skipped 1 previous similar message [ 5827.039130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5827.137668] Lustre: server umount lustre-OST0000 complete [ 5832.674300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5832.675157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5832.681430] Lustre: Skipped 2 previous similar messages [ 5835.168378] Lustre: server umount lustre-MDT0000 complete [ 5836.923834] LustreError: 436010:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771974357 with bad export cookie 10801686777709031736 [ 5836.929340] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5836.930751] LustreError: 436010:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5837.098240] Lustre: server umount lustre-MDT0001 complete [ 5840.916938] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5841.737851] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 18:06:01 (1771974361) [ 5844.956774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5845.203432] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5845.242125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5847.320636] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5850.594330] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5851.381793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5851.564654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5853.682251] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5857.265338] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5859.923799] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5863.112227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5863.295444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5865.377698] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5868.861176] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5869.775910] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5875.827287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5880.804497] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5880.806828] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5880.808314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5880.808323] Lustre: Skipped 1 previous similar message [ 5880.829173] Lustre: Skipped 1 previous similar message [ 5883.706934] Lustre: server umount lustre-OST0000 complete [ 5885.921327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5885.922374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5885.928764] Lustre: Skipped 1 previous similar message [ 5885.932393] Lustre: Skipped 1 previous similar message [ 5891.681688] Lustre: server umount lustre-MDT0000 complete [ 5893.475882] LustreError: 440600:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771974414 with bad export cookie 10801686777709032569 [ 5893.481553] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5893.482896] LustreError: 440600:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5899.668892] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 5901.221241] Key type lgssc unregistered [ 5901.375753] LNet: 444465:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5901.381233] LNet: Removed LNI 192.168.204.111@tcp [ 5901.834080] Key type .llcrypt unregistered [ 5901.836335] Key type ._llcrypt unregistered [ 5913.345653] alg: No test for adler32 (adler32-zlib) [ 5914.098244] Key type ._llcrypt registered [ 5914.099043] Key type .llcrypt registered [ 5914.144352] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 5914.269184] Key type .llcrypt unregistered [ 5914.271073] Key type ._llcrypt unregistered [ 5923.018887] alg: No test for adler32 (adler32-zlib) [ 5923.771251] Key type ._llcrypt registered [ 5923.772103] Key type .llcrypt registered [ 5923.817082] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 5924.183276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5924.445462] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 5924.521399] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 5924.522953] LNet: Accept secure, port 988 [ 5926.103126] Key type lgssc registered [ 5926.662553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5930.485758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5931.709913] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5931.732433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5933.872774] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5937.124400] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5938.019647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5938.172763] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5938.180812] LustreError: Skipped 1 previous similar message [ 5938.184337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5940.331852] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5943.269320] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5943.277690] LustreError: Skipped 1 previous similar message [ 5944.052524] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5946.348592] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 5947.408949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5947.591100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5949.746867] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 5958.116630] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5958.121517] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5958.130192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5958.134251] Lustre: Skipped 1 previous similar message [ 5963.235770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5963.239737] Lustre: Skipped 1 previous similar message [ 5963.845853] Lustre: server umount lustre-OST0000 complete [ 5967.843418] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5967.848644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5967.856401] Lustre: Skipped 1 previous similar message [ 5967.859790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5971.875986] Lustre: server umount lustre-MDT0000 complete [ 5973.473178] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5973.481581] LustreError: Skipped 3 previous similar messages [ 5973.710756] LustreError: 446771:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771974494 with bad export cookie 7254161223677118335 [ 5973.717600] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5973.717991] LustreError: 446771:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5973.898422] Lustre: server umount lustre-MDT0001 complete [ 5979.888518] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 5981.441494] Key type lgssc unregistered [ 5981.587499] LNet: 450055:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5981.592820] LNet: Removed LNI 192.168.204.111@tcp [ 5982.055575] Key type .llcrypt unregistered [ 5982.057536] Key type ._llcrypt unregistered [ 5993.946213] alg: No test for adler32 (adler32-zlib) [ 5994.699201] Key type ._llcrypt registered [ 5994.700088] Key type .llcrypt registered [ 5994.749891] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 5995.358663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5995.512764] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 5995.591227] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 5995.593195] LNet: Accept secure, port 988 [ 5997.183158] Key type lgssc registered [ 5997.773250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6001.672366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6018.207301] LustreError: 451490:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 1: rc = -5 [ 6049.951231] LustreError: 451490:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 4: rc = -110 [ 6079.647204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6081.818903] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 6085.458583] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6088.619844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6102.341660] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6102.376408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6102.419099] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [ 6104.474494] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 6107.618753] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6108.513079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6108.684963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6110.647948] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 6114.325988] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6117.175960] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6122.976239] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6122.981962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6122.991221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6124.002435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6124.002489] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6124.011544] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6126.201363] Lustre: server umount lustre-OST0000 complete [ 6129.120353] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6129.125226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6129.133118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6129.138459] Lustre: Skipped 1 previous similar message [ 6132.192646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6132.193474] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6134.173890] Lustre: server umount lustre-MDT0000 complete [ 6135.961534] LustreError: 451496:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771974656 with bad export cookie 9981053952515375721 [ 6135.967389] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6135.968591] LustreError: 451496:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6136.109668] Lustre: server umount lustre-MDT0001 complete [ 6141.997813] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 6143.402554] Key type lgssc unregistered [ 6143.546403] LNet: 455188:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6143.551242] LNet: Removed LNI 192.168.204.111@tcp [ 6143.980340] Key type .llcrypt unregistered [ 6143.982324] Key type ._llcrypt unregistered [ 6155.865471] alg: No test for adler32 (adler32-zlib) [ 6156.618236] Key type ._llcrypt registered [ 6156.619069] Key type .llcrypt registered [ 6156.669340] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 6157.088723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6157.356773] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 6157.424898] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 6157.426475] LNet: Accept secure, port 988 [ 6159.015147] Key type lgssc registered [ 6159.601636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6163.516138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6164.760520] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6164.781651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6166.911159] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 6170.082153] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6171.018788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6171.216268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6173.415825] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 6177.159288] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6179.941534] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6183.199448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6183.408540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6185.569224] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 6189.336858] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6194.669703] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [ 6196.704564] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6196.711780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6196.723195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6199.777897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6199.781253] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6199.784140] Lustre: Skipped 1 previous similar message [ 6199.792114] Lustre: Skipped 1 previous similar message [ 6201.895983] Lustre: server umount lustre-MDT0000 complete [ 6202.981187] LustreError: 456685:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.11@tcp arrived at 1771974723 with bad export cookie 17130482419121966335 [ 6203.946637] LustreError: 456643:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771974724 with bad export cookie 17130482419121965670 [ 6203.957662] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6203.958331] LustreError: 456643:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6204.896989] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6204.897946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6204.905231] LustreError: Skipped 1 previous similar message [ 6204.921097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6204.924945] Lustre: Skipped 2 previous similar messages [ 6208.928557] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6208.928670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6208.936762] LustreError: Skipped 1 previous similar message [ 6210.112890] Lustre: server umount lustre-OST0000 complete [ 6213.506855] Lustre: server umount lustre-MDT0001 complete [ 6219.795728] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 6221.337938] Key type lgssc unregistered [ 6221.490559] LNet: 460467:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6221.495849] LNet: Removed LNI 192.168.204.111@tcp [ 6221.943920] Key type .llcrypt unregistered [ 6221.946445] Key type ._llcrypt unregistered [ 6234.132125] alg: No test for adler32 (adler32-zlib) [ 6234.884241] Key type ._llcrypt registered [ 6234.885070] Key type .llcrypt registered [ 6234.928572] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 6235.422336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6235.571260] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 6235.652043] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 6235.653715] LNet: Accept secure, port 988 [ 6237.247139] Key type lgssc registered [ 6237.803312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6241.499796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6242.714907] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6242.738880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6244.846835] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 6247.906613] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6248.902392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6249.112912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6251.245091] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 6254.737149] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6257.462785] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6260.720064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6260.898347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6262.910337] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 6266.535521] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6274.221136] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:161 [ 6279.652692] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6279.655160] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6279.663511] Lustre: Skipped 1 previous similar message [ 6282.238287] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.11@tcp (stopping) [ 6282.242795] Lustre: Skipped 1 previous similar message [ 6284.566311] Lustre: server umount lustre-MDT0001 complete [ 6284.769325] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6284.780248] LustreError: Skipped 1 previous similar message [ 6289.890424] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6289.891133] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6289.891217] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6289.892440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6293.057666] Lustre: server umount lustre-OST0000 complete [ 6301.151283] Lustre: 464997:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771974815/real 1771974815] req@00000000b95d3f3a x1858050201760832/t0(0) o251->MGC192.168.204.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1771974821 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6301.266168] Lustre: server umount lustre-MDT0000 complete [ 6308.673575] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 6310.282223] Key type lgssc unregistered [ 6310.443672] LNet: 465752:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6310.448875] LNet: Removed LNI 192.168.204.111@tcp [ 6310.926943] Key type .llcrypt unregistered [ 6310.929970] Key type ._llcrypt unregistered [ 6323.176206] alg: No test for adler32 (adler32-zlib) [ 6323.928230] Key type ._llcrypt registered [ 6323.928993] Key type .llcrypt registered [ 6323.972638] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 6330.671845] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 6331.121209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6331.361340] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 6331.433079] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 6331.434605] LNet: Accept secure, port 988 [ 6333.023217] Key type lgssc registered [ 6333.613247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6338.129565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6341.413242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6344.427357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6347.579147] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6351.594455] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 6355.785873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6358.919207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6358.955059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6360.056582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6360.069102] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6360.124503] Lustre: lustre-MDT0000: new disk, initializing [ 6360.154591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6360.163260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6362.136944] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 6366.772769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6369.949540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6369.989379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6369.994202] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6370.111205] Lustre: lustre-OST0000: new disk, initializing [ 6370.114906] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6370.118320] Lustre: Skipped 1 previous similar message [ 6370.148729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6371.980319] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 6376.947542] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6386.738228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6389.940416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6389.980054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6390.024601] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6390.045905] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6390.084179] Lustre: lustre-MDT0001: new disk, initializing [ 6390.105111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6390.108150] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6390.127367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6392.183937] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 6397.021048] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 6397.146272] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6399.205370] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 6399.330239] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6405.118271] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.11@tcp (stopping) [ 6406.115178] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6406.115212] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6406.116666] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6406.131310] Lustre: Skipped 1 previous similar message [ 6407.764207] Lustre: server umount lustre-MDT0001 complete [ 6411.234760] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6411.243057] LustreError: Skipped 1 previous similar message [ 6416.354736] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6416.362361] LustreError: Skipped 1 previous similar message [ 6421.473697] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6421.474173] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6421.489402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6421.493280] Lustre: Skipped 1 previous similar message [ 6426.593376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6426.594254] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6431.712964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6431.713701] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6435.295129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6435.384305] Lustre: server umount lustre-OST0000 complete [ 6449.631156] Lustre: 474153:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771974964/real 1771974964] req@00000000e585a505 x1858050302427776/t0(0) o251->MGC192.168.204.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1771974970 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6449.737823] Lustre: server umount lustre-MDT0000 complete [ 6457.235085] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 6458.650171] Key type lgssc unregistered [ 6458.785475] LNet: 474910:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6458.790520] LNet: Removed LNI 192.168.204.111@tcp [ 6459.221525] Key type .llcrypt unregistered [ 6459.223895] Key type ._llcrypt unregistered [ 6478.394151] alg: No test for adler32 (adler32-zlib) [ 6479.147301] Key type ._llcrypt registered [ 6479.148141] Key type .llcrypt registered [ 6479.203964] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 6479.363234] Key type .llcrypt unregistered [ 6479.365229] Key type ._llcrypt unregistered [ 6487.625443] alg: No test for adler32 (adler32-zlib) [ 6488.377204] Key type ._llcrypt registered [ 6488.377976] Key type .llcrypt registered [ 6488.415654] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 6488.804785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6488.964212] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 6489.031317] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 6489.033068] LNet: Accept secure, port 988 [ 6490.623134] Key type lgssc registered [ 6491.224941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6495.342204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6498.520661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6501.530165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6504.790731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6504.824498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6505.939951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6505.954286] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6506.002969] Lustre: lustre-MDT0000: new disk, initializing [ 6506.036757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6506.049484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6508.226849] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 6513.819589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6513.855693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6513.907120] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6513.954173] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6513.956976] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6513.957870] Lustre: Skipped 1 previous similar message [ 6514.007638] Lustre: lustre-MDT0001: new disk, initializing [ 6514.054463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6514.073243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6514.077203] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6516.154291] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 6521.319455] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6524.049389] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6527.216636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6527.258658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6527.264457] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6527.384091] Lustre: lustre-OST0000: new disk, initializing [ 6527.386563] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6527.417063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6529.361642] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 6534.152265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6534.160863] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6534.475800] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6539.235497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6539.237000] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6539.243473] Lustre: Skipped 2 previous similar messages [ 6539.247053] Lustre: Skipped 1 previous similar message [ 6543.517038] Lustre: server umount lustre-MDT0000 complete [ 6544.355642] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6544.362411] LustreError: Skipped 2 previous similar messages [ 6545.281754] LustreError: 479252:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771975066 with bad export cookie 7233319099033600586 [ 6545.287157] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6545.288820] LustreError: 479252:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6545.449053] Lustre: server umount lustre-MDT0001 complete [ 6552.543186] Lustre: 481652:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771975067/real 1771975067] req@0000000097b68976 x1858050467052160/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1771975073 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6552.657339] Lustre: server umount lustre-OST0000 complete [ 6557.485066] Lustre: DEBUG MARKER: == conf-sanity test 73: failnode to update from mountdata properly ========================================================== 18:17:57 (1771975077) [ 6557.995078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6561.126367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6561.351688] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6561.390183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6563.415487] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 6566.880988] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6567.379701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6567.516745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6569.459528] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 6572.935629] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6575.541441] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6578.651826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6578.696271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6578.794416] Lustre: Found index 0 for lustre-OST0000, updating log [ 6578.867748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6580.728448] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 6584.223496] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6592.169910] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5 to 0x0:33 [ 6597.602654] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6597.603071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6597.608458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6597.614973] Lustre: Skipped 1 previous similar message [ 6597.618560] Lustre: Skipped 1 previous similar message [ 6598.775580] Lustre: server umount lustre-OST0000 complete [ 6602.721511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6602.722287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6602.729501] Lustre: Skipped 1 previous similar message [ 6602.738380] Lustre: Skipped 1 previous similar message [ 6608.864664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6608.870154] Lustre: Skipped 3 previous similar messages [ 6609.883714] Lustre: server umount lustre-MDT0000 complete [ 6611.629705] LustreError: 482458:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771975132 with bad export cookie 7233319099033601923 [ 6611.635675] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6611.636598] LustreError: 482458:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6615.648768] Lustre: DEBUG MARKER: == conf-sanity test 75: The order of --index should be irrelevant ========================================================== 18:18:55 (1771975135) [ 6618.336387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6620.583411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6623.030951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6625.265103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6632.744236] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 6636.754820] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 6641.361458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6644.557235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6647.475651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6650.493399] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6653.715823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6653.754948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6653.879028] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6653.894171] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6653.944497] Lustre: lustre-MDT0000: new disk, initializing [ 6653.975399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6653.986485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6656.068082] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 6661.561994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6661.604287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6661.656869] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6661.687331] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6661.691370] Lustre: Skipped 1 previous similar message [ 6661.742117] Lustre: lustre-MDT0001: new disk, initializing [ 6661.806794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6661.813106] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6663.940056] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 6668.992780] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6671.698460] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6674.902140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6674.944487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6674.950310] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6675.103546] Lustre: lustre-OST0000: new disk, initializing [ 6675.107094] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6677.264962] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all 8 [ 6681.608669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6681.614867] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6682.260767] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6684.426857] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6684.563669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6686.538915] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6686.662830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6691.810656] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6691.811038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6691.815805] LustreError: Skipped 1 previous similar message [ 6691.823923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6691.829608] Lustre: Skipped 1 previous similar message [ 6693.758468] Lustre: server umount lustre-OST0000 complete [ 6693.761477] Lustre: Skipped 1 previous similar message [ 6696.930794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6696.938416] Lustre: Skipped 2 previous similar messages [ 6702.049091] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6702.056800] LustreError: Skipped 1 previous similar message [ 6703.542868] LustreError: 489511:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771975224 with bad export cookie 7233319099033602980 [ 6703.548926] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6703.549769] LustreError: 489511:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6715.126829] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 6719.258930] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 6724.508626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6728.086398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6731.278460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6734.708145] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6735.589255] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6607 sec ========== 18:20:55 (1771975255) [ 6755.240627] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 6756.845466] Key type lgssc unregistered [ 6757.007469] LNet: 498245:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6757.012905] LNet: Removed LNI 192.168.204.111@tcp [ 6757.465644] Key type .llcrypt unregistered [ 6757.467772] Key type ._llcrypt unregistered