[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 458137486 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2512MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003103] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005008] kvm-guest: setup PV IPIs [ 0.007622] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010141] LSM: Security Framework initializing [ 0.011056] Yama: becoming mindful. [ 0.012047] SELinux: Initializing. [ 0.014057] *** VALIDATE selinux *** [ 0.022438] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027264] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029129] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030108] *** VALIDATE tmpfs *** [ 0.032241] *** VALIDATE proc *** [ 0.033254] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035264] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036157] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038029] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.042178] debug: unmapping init [mem 0xffffffff8de59000-0xffffffff8de60fff] [ 0.044953] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045700] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046019] ... version: 2 [ 0.046898] ... bit width: 48 [ 0.047017] ... generic registers: 4 [ 0.048013] ... value mask: 0000ffffffffffff [ 0.049016] ... max period: 00007fffffffffff [ 0.050014] ... fixed-purpose events: 3 [ 0.051012] ... event mask: 000000070000000f [ 0.052299] rcu: Hierarchical SRCU implementation. [ 0.054423] smp: Bringing up secondary CPUs ... [ 0.055649] x86: Booting SMP configuration: [ 0.056026] .... node #0, CPUs: #1 #2 #3 [ 0.059293] smp: Brought up 1 node, 4 CPUs [ 0.061013] smpboot: Max logical packages: 1 [ 0.062020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.096592] node 0 deferred pages initialised in 31ms [ 0.099491] devtmpfs: initialized [ 0.100217] x86/mm: Memory block size: 128MB [ 0.102479] gcov: version magic: 0x41383552 [ 0.104109] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.105072] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.106282] pinctrl core: initialized pinctrl subsystem [ 0.107188] [ 0.107518] ************************************************************* [ 0.108013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.109009] ** ** [ 0.110009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.111011] ** ** [ 0.112012] ** This means that this kernel is built to expose internal ** [ 0.113009] ** IOMMU data structures, which may compromise security on ** [ 0.114010] ** your system. ** [ 0.115009] ** ** [ 0.116009] ** If you see this message and you are not debugging the ** [ 0.117012] ** kernel, report this immediately to your vendor! ** [ 0.118009] ** ** [ 0.119013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.120016] ************************************************************* [ 0.121734] NET: Registered protocol family 16 [ 0.122429] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.123084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.124076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.125483] cpuidle: using governor menu [ 0.126000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.128507] PCI: Using configuration type 1 for base access [ 0.131144] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.140193] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.142079] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.145222] cryptd: max_cpu_qlen set to 1000 [ 0.148205] ACPI: Added _OSI(Module Device) [ 0.149012] ACPI: Added _OSI(Processor Device) [ 0.151015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.153014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.157890] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.165338] ACPI: Interpreter enabled [ 0.166063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.167011] ACPI: Using IOAPIC for interrupt routing [ 0.168099] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.169431] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.178000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.180048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.183020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.186104] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.191000] acpiphp: Slot [2] registered [ 0.193139] acpiphp: Slot [5] registered [ 0.194125] acpiphp: Slot [6] registered [ 0.196125] acpiphp: Slot [7] registered [ 0.197115] acpiphp: Slot [8] registered [ 0.199123] acpiphp: Slot [9] registered [ 0.200129] acpiphp: Slot [10] registered [ 0.201121] acpiphp: Slot [3] registered [ 0.203102] acpiphp: Slot [4] registered [ 0.204123] acpiphp: Slot [11] registered [ 0.206092] acpiphp: Slot [12] registered [ 0.207106] acpiphp: Slot [13] registered [ 0.209110] acpiphp: Slot [14] registered [ 0.210100] acpiphp: Slot [15] registered [ 0.211104] acpiphp: Slot [16] registered [ 0.213123] acpiphp: Slot [17] registered [ 0.214116] acpiphp: Slot [18] registered [ 0.216112] acpiphp: Slot [19] registered [ 0.217113] acpiphp: Slot [20] registered [ 0.219105] acpiphp: Slot [21] registered [ 0.220072] acpiphp: Slot [22] registered [ 0.221123] acpiphp: Slot [23] registered [ 0.222230] acpiphp: Slot [24] registered [ 0.224106] acpiphp: Slot [25] registered [ 0.226098] acpiphp: Slot [26] registered [ 0.227099] acpiphp: Slot [27] registered [ 0.229126] acpiphp: Slot [28] registered [ 0.230144] acpiphp: Slot [29] registered [ 0.231110] acpiphp: Slot [30] registered [ 0.233168] acpiphp: Slot [31] registered [ 0.234083] PCI host bridge to bus 0000:00 [ 0.236076] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.238041] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.240025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.243026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.245079] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.248030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.250177] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.252535] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.256218] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.264831] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.269055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.272018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.274017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.277025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.279548] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.282754] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.284044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.288624] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.293015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.306882] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.310995] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.317421] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.326020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.337020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.352019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.362267] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.370026] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.376019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.388019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.399460] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.404019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.409020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.422023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.430578] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.441016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.452024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.469020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.481337] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.488019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.492021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.505030] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.509921] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.512925] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.518016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.529016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.536937] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.539407] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.542363] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.544437] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.546241] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.551138] iommu: Default domain type: Passthrough [ 0.552377] SCSI subsystem initialized [ 0.553135] ACPI: bus type USB registered [ 0.554085] usbcore: registered new interface driver usbfs [ 0.555055] usbcore: registered new interface driver hub [ 0.556113] usbcore: registered new device driver usb [ 0.558178] pps_core: LinuxPPS API ver. 1 registered [ 0.559012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.562070] PTP clock support registered [ 0.564068] EDAC MC: Ver: 3.0.0 [ 0.565583] PCI: Using ACPI for IRQ routing [ 0.566722] NetLabel: Initializing [ 0.568013] NetLabel: domain hash size = 128 [ 0.569010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.571083] NetLabel: unlabeled traffic allowed by default [ 0.573166] vgaarb: loaded [ 0.574208] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.576015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.580420] clocksource: Switched to clocksource kvm-clock [ 0.693156] VFS: Disk quotas dquot_6.6.0 [ 0.694850] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.697366] *** VALIDATE ramfs *** [ 0.698637] *** VALIDATE hugetlbfs *** [ 0.700335] pnp: PnP ACPI init [ 0.702788] pnp: PnP ACPI: found 6 devices [ 0.720336] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.723801] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.725783] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.727958] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.730289] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.732487] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.734916] NET: Registered protocol family 2 [ 0.737089] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.741785] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.745803] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.751690] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.755967] TCP: Hash tables configured (established 65536 bind 65536) [ 0.759138] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.761691] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.763712] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.765637] NET: Registered protocol family 1 [ 0.767991] RPC: Registered named UNIX socket transport module. [ 0.770043] RPC: Registered udp transport module. [ 0.771608] RPC: Registered tcp transport module. [ 0.773193] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.775538] NET: Registered protocol family 44 [ 0.777025] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.779451] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.781299] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.783605] PCI: CLS 0 bytes, default 64 [ 0.785835] Unpacking initramfs... [ 2.277914] debug: unmapping init [mem 0xffff9cff3cc54000-0xffff9cff3ffbffff] [ 2.284624] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.286914] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.291263] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.806885] Initialise system trusted keyrings [ 2.808476] Key type blacklist registered [ 2.810792] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.819726] zbud: loaded [ 2.822778] *** VALIDATE nfs *** [ 2.823614] *** VALIDATE nfs4 *** [ 2.824653] pstore: using deflate compression [ 2.827422] Platform Keyring initialized [ 2.924050] NET: Registered protocol family 38 [ 2.925989] Key type asymmetric registered [ 2.927479] Asymmetric key parser 'x509' registered [ 2.929405] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.932251] io scheduler mq-deadline registered [ 2.933931] io scheduler kyber registered [ 2.935578] io scheduler bfq registered [ 2.937194] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.940111] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.942024] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.943773] ACPI: Power Button [PWRF] [ 2.947330] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.953315] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.962582] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.968062] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.988719] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.015729] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.044447] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.049522] Non-volatile memory driver v1.3 [ 3.050851] Linux agpgart interface v0.103 [ 3.086606] virtio_blk virtio1: [vda] 67992 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.090835] vda: detected capacity change from 0 to 34811904 [ 3.104375] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.107225] vdb: detected capacity change from 0 to 1073741824 [ 3.123459] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.126428] vdc: detected capacity change from 0 to 2621440000 [ 3.139718] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.142473] vdd: detected capacity change from 0 to 2621440000 [ 3.157308] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.159902] vde: detected capacity change from 0 to 4294967296 [ 3.174868] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.177654] vdf: detected capacity change from 0 to 4294967296 [ 3.190391] libphy: Fixed MDIO Bus: probed [ 3.199559] usbcore: registered new interface driver usbserial_generic [ 3.202350] usbserial: USB Serial support registered for generic [ 3.205124] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.209975] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.211971] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.214776] mousedev: PS/2 mouse device common for all mice [ 3.217153] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.219392] rtc_cmos 00:05: RTC can wake from S4 [ 3.224799] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.229123] rtc_cmos 00:05: registered as rtc0 [ 3.229636] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.231119] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.237911] intel_pstate: CPU model not supported [ 3.240992] hid: raw HID events driver (C) Jiri Kosina [ 3.243206] usbcore: registered new interface driver usbhid [ 3.245097] usbhid: USB HID core driver [ 3.246653] drop_monitor: Initializing network drop monitor service [ 3.249205] Initializing XFRM netlink socket [ 3.251255] NET: Registered protocol family 10 [ 3.254497] Segment Routing with IPv6 [ 3.256165] NET: Registered protocol family 17 [ 3.258320] mpls_gso: MPLS GSO support [ 3.263260] RAS: Correctable Errors collector initialized. [ 3.265432] AVX version of gcm_enc/dec engaged. [ 3.267136] AES CTR mode by8 optimization enabled [ 3.342684] sched_clock: Marking stable (3342650640, 0)->(4245668735, -903018095) [ 3.346607] registered taskstats version 1 [ 3.348536] Loading compiled-in X.509 certificates [ 3.350759] zswap: loaded using pool lzo/zbud [ 3.374294] Key type big_key registered [ 3.386674] Key type encrypted registered [ 3.388398] ima: No TPM chip found, activating TPM-bypass! [ 3.390301] ima: Allocated hash algorithm: sha1 [ 3.391644] ima: No architecture policies found [ 3.393408] evm: Initialising EVM extended attributes: [ 3.395143] evm: security.selinux [ 3.396346] evm: security.ima [ 3.397243] evm: security.capability [ 3.398621] evm: HMAC attrs: 0x1 [ 3.400819] rtc_cmos 00:05: setting system clock to 2026-04-14 20:44:25 UTC (1776199465) [ 3.406834] debug: unmapping init [mem 0xffffffff8ee03000-0xffffffff8effffff] [ 3.409348] debug: unmapping init [mem 0xffffffff8db82000-0xffffffff8de58fff] [ 3.419088] Write protecting the kernel read-only data: 28672k [ 3.422155] debug: unmapping init [mem 0xffffffff8c203000-0xffffffff8c3fffff] [ 3.425267] debug: unmapping init [mem 0xffffffff8cb14000-0xffffffff8cbfffff] [ 3.457276] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.464363] systemd[1]: Detected virtualization kvm. [ 3.465731] systemd[1]: Detected architecture x86-64. [ 3.467297] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.493192] systemd[1]: No hostname configured. [ 3.494746] systemd[1]: Set hostname to . [ 3.496531] random: systemd: uninitialized urandom read (16 bytes read) [ 3.498716] systemd[1]: Initializing machine ID from random generator. [ 3.545716] random: ln: uninitialized urandom read (6 bytes read) [ 3.646739] random: systemd: uninitialized urandom read (16 bytes read) [ 3.648869] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.652627] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.660145] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.252834] device-mapper: uevent: version 1.0.3 [ 4.255320] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.968683] virtio_net virtio0 ens2: renamed from eth0 [ 4.976383] random: fast init done [ 5.035657] scsi host0: ata_piix [ 5.055180] scsi host1: ata_piix [ 5.122636] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.124926] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.610101] dracut-initqueue[590]: RTNETLINK answers: File exists [ 9.903919] random: crng init done [ 9.904959] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.384098] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.376443] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.622439] SELinux: Disabled at runtime. [ 11.677499] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.683523] systemd[1]: Detected virtualization kvm. [ 11.684844] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.128576] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.131529] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.135332] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.140354] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.144925] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.151894] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.164605] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ 12.259510] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.579068] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.823073] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.854344] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.975939] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.991813] EDAC sbridge: Ver: 1.1.2 [ 14.402755] Key type dns_resolver registered [ 14.706189] NFS: Registering the id_resolver key type [ 14.708173] Key type id_resolver registered [ 14.709611] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg219-server login: [ 55.154056] libcfs: loading out-of-tree module taints kernel. [ 55.193205] alg: No test for adler32 (adler32-zlib) [ 55.946434] Key type ._llcrypt registered [ 55.949567] Key type .llcrypt registered [ 56.045829] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 72.450793] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 73.927659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 74.697973] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 75.702890] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 75.714891] LNet: Accept secure, port 988 [ 77.495881] Key type lgssc registered [ 79.226492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 94.185594] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 96.485612] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 104.637245] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 110.316363] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 115.841866] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 126.650179] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 128.054446] hrtimer: interrupt took 3448970 ns [ 135.692732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 135.760409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 136.973189] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 136.998275] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 137.083547] Lustre: lustre-MDT0000: new disk, initializing [ 137.214880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 137.228499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 141.112730] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 152.762654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 152.848615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 152.899326] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 152.922331] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 152.931088] Lustre: Skipped 1 previous similar message [ 152.998446] Lustre: lustre-MDT0001: new disk, initializing [ 153.053962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 153.071327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 153.080844] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 156.401100] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 169.057655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 169.158392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 169.166587] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 169.422259] Lustre: lustre-OST0000: new disk, initializing [ 169.433405] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 169.506964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 170.259105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 170.267392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 173.637257] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 186.459723] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 186.546605] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 186.559691] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 186.663783] Lustre: lustre-OST0001: new disk, initializing [ 186.669569] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 186.726763] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 190.866317] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 192.579897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 192.587653] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 201.176778] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 206.553698] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 215.547645] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing check_logdir /tmp/testlogs/ [ 221.312224] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing yml_node [ 225.138357] Lustre: DEBUG MARKER: Client: 2.15.8.2 [ 227.550463] Lustre: DEBUG MARKER: MDS: 2.15.8.2 [ 230.167273] Lustre: DEBUG MARKER: OSS: 2.15.8.2 [ 232.059915] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Apr 14 16:48:13 EDT 2026 [ 240.257754] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball 110 [ 241.626586] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 259.046230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 259.047419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 259.059425] Lustre: Skipped 3 previous similar messages [ 259.071585] Lustre: Skipped 3 previous similar messages [ 264.164545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 264.171972] Lustre: Skipped 3 previous similar messages [ 269.280105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 269.288355] Lustre: Skipped 2 previous similar messages [ 271.327609] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 271.452173] Lustre: server umount lustre-MDT0000 complete [ 274.402430] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 274.421709] LustreError: Skipped 3 previous similar messages [ 274.680408] LustreError: 6246:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199736 with bad export cookie 12522886122970587443 [ 274.683761] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 274.691935] LustreError: 6246:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 279.520122] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 279.522551] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 279.522621] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 279.522626] Lustre: Skipped 1 previous similar message [ 279.533470] Lustre: Skipped 1 previous similar message [ 279.570755] LustreError: Skipped 1 previous similar message [ 280.614602] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 280.634667] LustreError: Skipped 1 previous similar message [ 281.102334] Lustre: server umount lustre-MDT0001 complete [ 284.718767] Lustre: server umount lustre-OST0000 complete [ 288.151918] Lustre: server umount lustre-OST0001 complete [ 294.495495] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 301.476965] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 310.202446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 315.266860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 320.416540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 326.300948] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 333.212562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 333.262703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 333.526575] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 333.567957] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 333.666712] Lustre: lustre-MDT0000: new disk, initializing [ 333.756964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 333.775763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 338.420325] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 350.502218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 350.627061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 350.838608] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 350.919828] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 350.929143] Lustre: Skipped 1 previous similar message [ 351.086464] Lustre: lustre-MDT0001: new disk, initializing [ 351.241044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 351.294251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 351.305141] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 356.493197] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 367.205623] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 372.653301] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 378.886990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 378.995050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 379.002725] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 379.208782] Lustre: lustre-OST0000: new disk, initializing [ 379.212835] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 379.291604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 380.800737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 380.810611] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 383.085688] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 391.580152] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 395.705743] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 395.910929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 399.547560] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 399.691527] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 401.379303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 401.382796] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 401.392229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 401.407578] Lustre: Skipped 4 previous similar messages [ 407.450528] Lustre: server umount lustre-OST0000 complete [ 411.619609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 411.621294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 411.628335] Lustre: Skipped 2 previous similar messages [ 411.640053] Lustre: Skipped 2 previous similar messages [ 417.223047] Lustre: server umount lustre-MDT0000 complete [ 421.866724] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 421.894548] LustreError: Skipped 1 previous similar message [ 422.594157] LustreError: 14331:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199884 with bad export cookie 12522886122970588906 [ 422.613498] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 422.619299] LustreError: 14331:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 429.979509] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 16:51:30 (1776199890) [ 433.035657] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 434.699668] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 16:51:35 (1776199895) [ 441.415708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 441.995805] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 442.046803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 448.156914] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 452.577512] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 452.598281] LustreError: Skipped 2 previous similar messages [ 456.892932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 461.319524] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 469.318705] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 475.015150] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 483.538994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 489.452210] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 498.066189] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 507.880115] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 507.904576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 507.971506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 507.973708] Lustre: Skipped 2 previous similar messages [ 508.903953] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 513.368081] Lustre: server umount lustre-OST0000 complete [ 513.371254] Lustre: Skipped 1 previous similar message [ 518.638856] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 518.650565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 518.668754] Lustre: Skipped 1 previous similar message [ 524.262310] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 524.287527] LustreError: Skipped 1 previous similar message [ 527.171421] LustreError: 18869:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199989 with bad export cookie 12522886122970589760 [ 527.178796] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 538.217604] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 540.577259] Key type lgssc unregistered [ 540.821625] LNet: 22544:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 540.829298] LNet: Removed LNI 192.168.202.119@tcp [ 541.542487] Key type .llcrypt unregistered [ 541.544533] Key type ._llcrypt unregistered [ 559.769884] alg: No test for adler32 (adler32-zlib) [ 560.525483] Key type ._llcrypt registered [ 560.527788] Key type .llcrypt registered [ 560.610392] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 561.385698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 561.597185] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 561.721440] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 561.724556] LNet: Accept secure, port 988 [ 563.351135] Key type lgssc registered [ 564.327725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 570.974936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 572.349651] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.391584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 575.717661] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 577.510728] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 582.575817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 582.624818] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 582.868511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 586.897274] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 593.945724] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 599.672992] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 605.589804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 605.977624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 610.779776] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 620.736585] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 622.408752] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 637.930305] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 637.931810] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 637.946572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 637.952543] Lustre: Skipped 1 previous similar message [ 637.967797] Lustre: Skipped 1 previous similar message [ 640.660154] Lustre: server umount lustre-OST0000 complete [ 644.577214] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 644.588530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 644.634037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 648.167365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 648.168972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 650.854814] Lustre: server umount lustre-MDT0000 complete [ 653.282278] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 653.296693] LustreError: Skipped 1 previous similar message [ 654.323465] LustreError: 23986:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200116 with bad export cookie 14898318951520972997 [ 654.331247] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 654.341466] LustreError: 23986:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 654.701942] Lustre: server umount lustre-MDT0001 complete [ 666.267240] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 668.590454] Key type lgssc unregistered [ 668.789274] LNet: 27695:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.799176] LNet: Removed LNI 192.168.202.119@tcp [ 669.520588] Key type .llcrypt unregistered [ 669.526253] Key type ._llcrypt unregistered [ 688.285320] alg: No test for adler32 (adler32-zlib) [ 689.042733] Key type ._llcrypt registered [ 689.045395] Key type .llcrypt registered [ 689.155558] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 690.273604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 690.694590] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 690.895951] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 690.900267] LNet: Accept secure, port 988 [ 692.519189] Key type lgssc registered [ 693.331581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 700.432241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 701.908666] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 701.970810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 705.936694] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 707.055809] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 712.169468] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 714.077642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 714.369214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 718.258310] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 724.845558] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 730.032744] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 735.760180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 735.978120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 739.640658] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 747.296639] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 752.371833] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 757.735921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 757.749427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 757.753186] Lustre: Skipped 2 previous similar messages [ 757.776032] Lustre: Skipped 2 previous similar messages [ 758.431278] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 759.246042] Lustre: server umount lustre-MDT0000 complete [ 761.826746] LustreError: 29142:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200223 with bad export cookie 1624496186487515658 [ 761.831254] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 761.833333] LustreError: 29142:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 762.854474] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 762.854785] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 762.874177] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 764.148736] Lustre: server umount lustre-MDT0001 complete [ 784.736418] Lustre: server umount lustre-OST0000 complete [ 799.748388] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 801.969155] Key type lgssc unregistered [ 802.290684] LNet: 33140:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.305424] LNet: Removed LNI 192.168.202.119@tcp [ 802.992616] Key type .llcrypt unregistered [ 802.995419] Key type ._llcrypt unregistered [ 824.954059] alg: No test for adler32 (adler32-zlib) [ 825.709403] Key type ._llcrypt registered [ 825.710791] Key type .llcrypt registered [ 825.837079] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 826.617540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 827.190845] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 827.350104] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 827.356332] LNet: Accept secure, port 988 [ 829.079571] Key type lgssc registered [ 829.952473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 837.486676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 838.920154] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 838.995691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 843.373267] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 844.259516] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 849.377262] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 852.315432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 852.747628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 857.563686] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 864.887444] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 870.003643] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 876.862763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 877.201374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 880.702319] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 887.873266] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 889.520942] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 898.024941] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 898.031756] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 898.054653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 898.718074] Lustre: lustre-OST0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 900.065324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 900.072796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 900.093059] Lustre: Skipped 1 previous similar message [ 903.706193] Lustre: server umount lustre-OST0000 complete [ 924.643942] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 924.656367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 924.670564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 926.689883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 928.219397] Lustre: server umount lustre-MDT0000 complete [ 931.760564] LustreError: 34581:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200393 with bad export cookie 6634840788195013138 [ 931.766214] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 931.781424] LustreError: 34581:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 938.258871] Lustre: server umount lustre-MDT0001 complete [ 950.611704] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 953.086711] Key type lgssc unregistered [ 953.290685] LNet: 38435:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.304981] LNet: Removed LNI 192.168.202.119@tcp [ 953.978432] Key type .llcrypt unregistered [ 953.982630] Key type ._llcrypt unregistered [ 980.311416] alg: No test for adler32 (adler32-zlib) [ 981.062787] Key type ._llcrypt registered [ 981.066672] Key type .llcrypt registered [ 981.213170] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 983.391170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 983.766328] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 983.974682] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 983.978455] LNet: Accept secure, port 988 [ 985.631386] Key type lgssc registered [ 987.354854] Lustre: Echo OBD driver; http://www.lustre.org/ [ 996.953146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 998.482402] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 998.595546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1004.001962] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1004.400820] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 1009.123332] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1013.215531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1013.568483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1017.895124] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 1025.496975] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1030.619157] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1037.002381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1037.383881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1041.946866] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 1049.431493] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1049.707647] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 1054.542840] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1058.053718] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1066.491293] LNetError: 120-3: Refusing connection from 192.168.202.119 for 127.0.0.2@tcp: No matching NI [ 1066.498868] LNetError: 39217:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1066.510063] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1070.195637] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-04-1417h02m10s [ 1071.703741] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1075.169090] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1075.178684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1075.189648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1075.680551] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1075.687893] LNetError: 120-3: Refusing connection from 192.168.202.119 for 127.0.0.2@tcp: No matching NI [ 1075.688173] Lustre: Skipped 1 previous similar message [ 1075.697479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1075.699062] LNetError: 39218:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1075.699145] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1075.744958] Lustre: Skipped 1 previous similar message [ 1076.907709] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 1079.763732] Lustre: server umount lustre-MDT0000 complete [ 1080.805267] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1080.827857] LustreError: Skipped 2 previous similar messages [ 1080.836884] LNetError: 120-3: Refusing connection from 192.168.202.119 for 127.0.0.2@tcp: No matching NI [ 1080.859664] LNetError: 39219:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1080.870280] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1082.937888] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1085.921309] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1085.924635] LNetError: 120-3: Refusing connection from 192.168.202.119 for 127.0.0.2@tcp: No matching NI [ 1085.929890] LustreError: Skipped 3 previous similar messages [ 1085.967908] LNetError: 39220:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1085.973938] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1086.944131] Lustre: 39232:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776200542/real 1776200542] req@00000000c509b67c x1862481158883840/t0(0) o400->MGC192.168.202.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776200549 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1086.959451] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1089.060929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1093.089709] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0x422e0fe2ca82bdb3 to 0x422e0fe2ca82c20c [ 1093.106184] Lustre: MGC192.168.202.119@tcp: Connection restored to 192.168.202.119@tcp (at 0@lo) [ 1093.350024] LNetError: 120-3: Refusing connection from 192.168.202.119 for 127.0.0.2@tcp: No matching NI [ 1093.354550] LNetError: 39217:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1093.363156] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1093.369179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1093.444993] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 1097.389800] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 1098.737949] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1098.751351] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 1098.771838] Lustre: Skipped 2 previous similar messages [ 1100.993403] Lustre: DEBUG MARKER: Wait for df (28903) ... [ 1103.040989] Lustre: DEBUG MARKER: done [ 1108.980296] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1108.990241] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1108.996121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1109.007929] Lustre: Skipped 1 previous similar message [ 1112.782661] Lustre: server umount lustre-OST0000 complete [ 1119.200817] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1119.203274] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1119.214533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1119.230556] Lustre: Skipped 2 previous similar messages [ 1119.238853] LNetError: 120-3: Refusing connection from 192.168.202.119 for 127.0.0.2@tcp: No matching NI [ 1119.247531] LNetError: 39218:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1119.257673] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1122.910631] Lustre: server umount lustre-MDT0000 complete [ 1124.322324] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1124.337877] LustreError: Skipped 12 previous similar messages [ 1127.183090] LustreError: 42628:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200589 with bad export cookie 4768766522166526476 [ 1127.186252] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1127.189858] LustreError: 42628:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1127.644826] Lustre: server umount lustre-MDT0001 complete [ 1143.023571] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 1148.027356] Key type lgssc unregistered [ 1148.409891] LNet: 45039:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1148.439353] LNet: Removed LNI 192.168.202.119@tcp [ 1149.616107] Key type .llcrypt unregistered [ 1149.617635] Key type ._llcrypt unregistered [ 1183.928232] alg: No test for adler32 (adler32-zlib) [ 1184.680362] Key type ._llcrypt registered [ 1184.682784] Key type .llcrypt registered [ 1184.774655] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 1197.234982] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 1198.329742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1198.620807] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 1198.818867] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 1198.823399] LNet: Accept secure, port 988 [ 1200.488839] Key type lgssc registered [ 1201.464218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1210.075394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1217.058316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1222.484985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1228.132852] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1234.409953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1234.491313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1235.776664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1235.807811] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1235.868972] Lustre: lustre-MDT0000: new disk, initializing [ 1235.942528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1235.960863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1240.530635] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 1250.488627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1250.587791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1250.707524] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1250.795752] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1250.806473] Lustre: Skipped 1 previous similar message [ 1250.875974] Lustre: lustre-MDT0001: new disk, initializing [ 1250.953261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1250.991817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1251.008795] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1256.804418] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 1267.742195] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1273.162601] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1280.075685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1280.148914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1280.153798] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1280.355868] Lustre: lustre-OST0000: new disk, initializing [ 1280.359850] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1280.435095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1285.274577] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 1286.255824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1286.271805] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1295.110301] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1299.194126] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1299.402536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1303.605818] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1303.795216] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1307.104435] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1307.110724] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1307.127577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1307.140114] Lustre: Skipped 1 previous similar message [ 1307.157738] Lustre: Skipped 1 previous similar message [ 1311.728368] Lustre: server umount lustre-OST0000 complete [ 1317.353751] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1317.360169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1317.371283] Lustre: Skipped 1 previous similar message [ 1317.388459] Lustre: Skipped 1 previous similar message [ 1321.570522] Lustre: server umount lustre-MDT0000 complete [ 1322.466473] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1322.487560] LustreError: Skipped 1 previous similar message [ 1323.495492] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1323.519127] LustreError: Skipped 1 previous similar message [ 1325.242953] LustreError: 49131:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200787 with bad export cookie 2631814826270632324 [ 1325.260620] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1325.262064] LustreError: 49131:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1325.531563] Lustre: server umount lustre-MDT0001 complete [ 1334.252950] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 17:06:35 (1776200795) [ 1341.358271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1341.741839] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1341.788844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1346.017529] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 1347.047098] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1352.169105] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1354.218351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1354.558138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1358.500104] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 1365.091606] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1370.260714] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1377.158238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1377.540418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1381.924683] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 1390.881822] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1397.749911] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-04-1417h07m38s [ 1399.608799] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1407.132880] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1410.074606] LNetError: 120-3: Refusing connection from 192.168.202.119 for 127.0.0.2@tcp: No matching NI [ 1410.087261] LNetError: 47348:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1410.099628] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1410.107158] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1410.176642] LustreError: 53685:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1430.732759] Lustre: lustre-MDT0000: Client 17d8d02c-1e1d-4dec-9010-b1ebf73a3fd3 (at 192.168.202.19@tcp) reconnecting [ 1450.215105] LustreError: 53685:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 awake [ 1453.716358] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1457.200805] Lustre: DEBUG MARKER: done [ 1466.848482] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1466.857117] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1466.877578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1468.901218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1468.915796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1472.074236] Lustre: server umount lustre-OST0000 complete [ 1477.601602] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1477.607036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1477.626174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1477.629500] Lustre: Skipped 1 previous similar message [ 1479.146286] LNetError: 120-3: Refusing connection from 192.168.202.119 for 127.0.0.2@tcp: No matching NI [ 1479.152388] LNetError: 47346:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1479.159772] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1482.513492] Lustre: server umount lustre-MDT0000 complete [ 1484.257866] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1484.270914] LNetError: 120-3: Refusing connection from 192.168.202.119 for 127.0.0.2@tcp: No matching NI [ 1484.280294] LNetError: 47347:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1484.285698] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1485.587191] LustreError: 53670:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200947 with bad export cookie 2631814826270633178 [ 1485.591822] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1485.595031] LustreError: 53670:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1496.656788] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 1499.070549] Key type lgssc unregistered [ 1499.343726] LNet: 58113:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1499.351930] LNet: Removed LNI 192.168.202.119@tcp [ 1500.023571] Key type .llcrypt unregistered [ 1500.026857] Key type ._llcrypt unregistered [ 1529.579287] alg: No test for adler32 (adler32-zlib) [ 1530.342774] Key type ._llcrypt registered [ 1530.352686] Key type .llcrypt registered [ 1530.518643] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 1546.388961] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 1547.900890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1548.445490] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 1548.637137] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 1548.642292] LNet: Accept secure, port 988 [ 1550.327124] Key type lgssc registered [ 1551.273528] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1559.399895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1565.642753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1572.027985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1578.494254] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1586.559264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1586.755738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1588.123734] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1588.149267] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1588.192434] Lustre: lustre-MDT0000: new disk, initializing [ 1588.280827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1588.304509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1593.788062] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 1604.172617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1604.266951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1604.374761] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1604.432646] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1604.436563] Lustre: Skipped 1 previous similar message [ 1604.543540] Lustre: lustre-MDT0001: new disk, initializing [ 1604.633579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1604.674072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1604.684260] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1608.943750] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 1619.050581] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1625.270317] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1632.292682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1632.364875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1632.370825] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1632.670489] Lustre: lustre-OST0000: new disk, initializing [ 1632.686052] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1632.793901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1637.017025] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 1641.077860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1641.099759] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1645.812132] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1649.950757] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1650.164906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1654.231313] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1654.555946] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1656.805241] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1656.808827] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1656.811306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1656.821076] LustreError: Skipped 1 previous similar message [ 1656.854871] Lustre: Skipped 1 previous similar message [ 1661.921569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1661.933525] Lustre: Skipped 2 previous similar messages [ 1662.407012] Lustre: server umount lustre-OST0000 complete [ 1667.043402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1667.053067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1667.063318] Lustre: Skipped 1 previous similar message [ 1667.080619] Lustre: Skipped 1 previous similar message [ 1672.168170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1672.182201] Lustre: Skipped 1 previous similar message [ 1672.551140] Lustre: server umount lustre-MDT0000 complete [ 1676.179151] LustreError: 62226:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201138 with bad export cookie 10047673596564610661 [ 1676.186124] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1676.188745] LustreError: 62226:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1676.565991] Lustre: server umount lustre-MDT0001 complete [ 1686.572071] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 17:12:27 (1776201147) [ 1688.703915] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1690.556975] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 17:12:31 (1776201151) [ 1702.661300] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 1706.130978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1706.552804] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1706.593662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1707.454308] Lustre: Failing over lustre-MDT0000 [ 1707.462024] LustreError: 67266:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1707.471025] LustreError: 67266:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1707.481812] LustreError: 67266:0:(lod_dev.c:437:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 1713.631993] Lustre: 67317:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201169/real 1776201169] req@00000000a15a82bb x1862481750282560/t0(0) o251->MGC192.168.202.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776201175 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1713.836360] Lustre: server umount lustre-MDT0000 complete [ 1719.334924] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 17:13:00 (1776201180) [ 1724.685326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1725.282975] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1725.326248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1729.707890] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 1730.537272] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1735.663695] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1737.128691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1737.437231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1741.192574] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 1747.259504] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1752.032362] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1758.001591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1758.263629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1761.896806] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 1768.790674] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1783.822966] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1790.947925] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1790.951687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1790.959480] Lustre: Skipped 2 previous similar messages [ 1790.967774] Lustre: Skipped 2 previous similar messages [ 1796.230723] Lustre: server umount lustre-MDT0000 complete [ 1799.094793] LustreError: 67863:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201261 with bad export cookie 10047673596564611788 [ 1799.108478] LustreError: 67863:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1799.123201] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1801.911439] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1805.312986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1808.287089] Lustre: 60451:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201263/real 1776201263] req@00000000a7a5199e x1862481750301888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776201270 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1808.321745] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1811.425094] Lustre: 60453:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201266/real 1776201266] req@000000007224b64f x1862481750301952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776201273 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1812.033796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1816.550663] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0x8b7081e6b64e22cc to 0x8b7081e6b64e2a5f [ 1816.575079] Lustre: MGC192.168.202.119@tcp: Connection restored to 192.168.202.119@tcp (at 0@lo) [ 1817.043773] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1817.159416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1818.158210] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1821.306929] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 1825.264653] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1825.286985] LustreError: Skipped 4 previous similar messages [ 1833.118840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1833.770076] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:65 [ 1838.098442] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 1839.081643] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1845.144702] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1850.610778] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1858.428850] Lustre: server umount lustre-MDT0000 complete [ 1858.431140] Lustre: Skipped 1 previous similar message [ 1859.041401] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1859.048157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1859.058414] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1859.066154] LustreError: Skipped 2 previous similar messages [ 1862.310784] LustreError: 67864:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201324 with bad export cookie 10047673596564613727 [ 1862.320470] LustreError: 67864:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1862.324680] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1869.310648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1871.775388] Lustre: 60451:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201326/real 1776201326] req@000000006a4ca25f x1862481750316416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776201333 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1871.802607] Lustre: 60451:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1871.807958] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1871.826387] Lustre: Skipped 2 previous similar messages [ 1876.895153] Lustre: 60451:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201331/real 1776201331] req@0000000035ea1c36 x1862481750316672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776201338 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1876.943124] Lustre: 60451:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1879.396786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1895.402155] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0x8b7081e6b64e2a5f to 0x8b7081e6b64e3078 [ 1895.412584] Lustre: MGC192.168.202.119@tcp: Connection restored to (at 0@lo) [ 1895.634138] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1895.655596] LustreError: Skipped 2 previous similar messages [ 1895.723720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1895.731114] Lustre: Skipped 1 previous similar message [ 1895.769773] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 1899.517782] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 1906.089564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1906.475273] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:97 [ 1910.616799] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 1911.795101] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1917.317889] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1922.649770] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1932.256735] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1932.266456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1932.278081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1932.285894] Lustre: Skipped 3 previous similar messages [ 1935.404849] Lustre: server umount lustre-MDT0000 complete [ 1935.411240] Lustre: Skipped 1 previous similar message [ 1939.118785] LustreError: 67865:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201401 with bad export cookie 10047673596564615288 [ 1939.119953] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1939.133389] LustreError: 67865:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1946.419439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1949.576698] Lustre: DEBUG MARKER: files compared the same [ 1950.623330] Lustre: 60451:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201405/real 1776201405] req@000000008cb5482e x1862481750329984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776201412 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1968.494325] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 1971.173613] Key type lgssc unregistered [ 1971.569761] LNet: 77943:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1971.585526] LNet: Removed LNI 192.168.202.119@tcp [ 1972.292646] Key type .llcrypt unregistered [ 1972.295843] Key type ._llcrypt unregistered [ 1995.667869] alg: No test for adler32 (adler32-zlib) [ 1996.430483] Key type ._llcrypt registered [ 1996.434619] Key type .llcrypt registered [ 1996.540947] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 1997.585577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1998.208213] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 1998.625769] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 1998.631472] LNet: Accept secure, port 988 [ 2000.375277] Key type lgssc registered [ 2002.294705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2010.541337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2011.991503] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2012.042671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2015.936346] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug +malloc all 8 [ 2017.249237] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2022.378577] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2024.338584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2024.553496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2028.911390] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug +malloc all 8 [ 2035.650446] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2040.737216] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2046.481246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2046.720116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2051.292922] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug +malloc all 8 [ 2053.926406] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:129 [ 2060.114934] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2063.168622] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 2078.688214] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2078.697370] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2078.717613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2078.739398] Lustre: Skipped 1 previous similar message [ 2082.148730] Lustre: server umount lustre-OST0000 complete [ 2088.933320] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2088.970836] Lustre: Skipped 1 previous similar message [ 2088.975931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2088.982389] Lustre: Skipped 1 previous similar message [ 2092.726944] Lustre: server umount lustre-MDT0000 complete [ 2094.047797] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2094.060833] LustreError: Skipped 1 previous similar message [ 2096.189401] LustreError: 79408:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201558 with bad export cookie 6909163343106998612 [ 2096.197605] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2096.445595] Lustre: server umount lustre-MDT0001 complete [ 2109.249607] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 2112.539829] Key type lgssc unregistered [ 2112.996317] LNet: 83106:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2113.005635] LNet: Removed LNI 192.168.202.119@tcp [ 2113.866962] Key type .llcrypt unregistered [ 2113.875533] Key type ._llcrypt unregistered [ 2138.617191] alg: No test for adler32 (adler32-zlib) [ 2139.376781] Key type ._llcrypt registered [ 2139.380982] Key type .llcrypt registered [ 2139.510719] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2140.465705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2140.899554] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 2141.125449] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 2141.130994] LNet: Accept secure, port 988 [ 2142.775150] Key type lgssc registered [ 2143.519237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2150.885927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2168.479421] LustreError: 84540:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.119@tcp: failed processing log, type 1: rc = -5 [ 2199.199426] LustreError: 84540:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.119@tcp: failed processing log, type 4: rc = -110 [ 2229.023424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2233.750334] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 2242.021930] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2249.626515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2249.714810] Lustre: *** cfs_fail_loc=706, val=0*** [ 2251.765239] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2251.832969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2251.902251] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 2256.370388] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 2256.868105] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2261.995170] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2264.640654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2264.984970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2265.021240] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:161 [ 2269.963423] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 2276.867695] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2281.786642] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2284.528649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2284.544218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2289.639198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2289.650968] Lustre: Skipped 1 previous similar message [ 2289.873077] Lustre: server umount lustre-OST0000 complete [ 2294.752984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2294.763914] Lustre: Skipped 1 previous similar message [ 2294.767525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2294.770404] Lustre: Skipped 1 previous similar message [ 2299.806702] Lustre: server umount lustre-MDT0000 complete [ 2299.874783] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2299.910876] LustreError: Skipped 1 previous similar message [ 2303.245816] LustreError: 87476:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201765 with bad export cookie 16917792450170011768 [ 2303.248901] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2303.256371] LustreError: 87476:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2303.468034] Lustre: server umount lustre-MDT0001 complete [ 2313.316634] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 2316.093220] Key type lgssc unregistered [ 2316.438933] LNet: 88280:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2316.459040] LNet: Removed LNI 192.168.202.119@tcp [ 2317.422185] Key type .llcrypt unregistered [ 2317.430200] Key type ._llcrypt unregistered [ 2338.477756] alg: No test for adler32 (adler32-zlib) [ 2339.234593] Key type ._llcrypt registered [ 2339.244068] Key type .llcrypt registered [ 2339.401885] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2340.473844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2340.761937] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 2340.982965] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 2340.987746] LNet: Accept secure, port 988 [ 2342.647125] Key type lgssc registered [ 2343.488974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2352.522166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2359.675290] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 2368.607219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2369.359813] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2369.549260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2374.627593] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2374.643096] LustreError: Skipped 2 previous similar messages [ 2375.052740] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 2379.745505] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2379.777037] LustreError: Skipped 1 previous similar message [ 2384.878892] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2384.917243] LustreError: Skipped 1 previous similar message [ 2385.354748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2385.546507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2389.826512] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 2391.014587] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2391.040672] LustreError: Skipped 5 previous similar messages [ 2397.778114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2397.859940] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:193 [ 2399.908128] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:193 [ 2402.643865] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 2415.594745] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2415.597499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.630179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2415.659645] Lustre: Skipped 1 previous similar message [ 2420.628286] Lustre: server umount lustre-OST0000 complete [ 2425.824913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2425.842634] Lustre: Skipped 1 previous similar message [ 2425.860739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2430.734290] Lustre: server umount lustre-MDT0000 complete [ 2430.949235] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2430.965132] LustreError: Skipped 6 previous similar messages [ 2435.227661] Lustre: server umount lustre-MDT0001 complete [ 2446.303969] Lustre: 93148:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201902/real 1776201902] req@00000000c1f93925 x1862482581796736/t0(0) o251->MGC192.168.202.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776201908 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2446.374304] Lustre: server umount MGS complete [ 2457.982085] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 17:25:19 (1776201919) [ 2488.078786] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 2499.338937] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2511.120845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2517.717235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2524.031291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2530.893964] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2538.227673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2538.320963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2543.380755] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 2552.851534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2552.904777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2553.059209] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2553.087477] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2553.168306] Lustre: lustre-MDT0001: new disk, initializing [ 2553.276226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2553.310791] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2553.321526] LustreError: 97853:0:(lod_dev.c:437:lod_sub_recovery_thread()) lustre-MDT0001-osd get update log failed: rc = -115 [ 2557.697823] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 2567.987976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2568.071632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2568.083698] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2568.299890] Lustre: lustre-OST0000: new disk, initializing [ 2568.303887] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2568.392265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2573.488337] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 2574.890376] Lustre: 99001:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@00000000f009ee56 x1862482581801344/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1776202048 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 2584.598866] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2593.480707] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2593.523763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2593.666588] Lustre: lustre-MDT0000: new disk, initializing [ 2593.850527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2593.901783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2594.274111] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2594.980390] Lustre: 99705:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@00000000649cfff4 x1862482581805184/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1776202069 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 2599.604451] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 2601.464203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2601.473590] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2604.068534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2604.082710] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2629.601769] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2629.616093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2629.627133] Lustre: Skipped 1 previous similar message [ 2629.633604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2629.637437] Lustre: Skipped 1 previous similar message [ 2635.735340] Lustre: server umount lustre-OST0000 complete [ 2641.377324] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2641.379140] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2641.392282] Lustre: Skipped 1 previous similar message [ 2641.401654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2641.425390] Lustre: Skipped 3 previous similar messages [ 2645.497310] Lustre: server umount lustre-MDT0000 complete [ 2646.496076] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2646.508032] LustreError: Skipped 4 previous similar messages [ 2663.392108] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2663.832274] Lustre: server umount lustre-MDT0001 complete [ 2673.120574] Lustre: 100904:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202129/real 1776202129] req@00000000bd3aa86b x1862482581816768/t0(0) o251->MGC192.168.202.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776202135 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2688.025275] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 17:29:08 (1776202148) [ 2706.739217] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 2709.272887] Key type lgssc unregistered [ 2709.559636] LNet: 102316:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2709.568506] LNet: Removed LNI 192.168.202.119@tcp [ 2710.472313] Key type .llcrypt unregistered [ 2710.474371] Key type ._llcrypt unregistered [ 2726.581896] alg: No test for adler32 (adler32-zlib) [ 2727.342563] Key type ._llcrypt registered [ 2727.347349] Key type .llcrypt registered [ 2727.458421] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2728.703332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2729.367321] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 2729.660444] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 2729.664956] LNet: Accept secure, port 988 [ 2731.311291] Key type lgssc registered [ 2732.449741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2739.625559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2741.142886] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2741.228381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2746.284255] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 2746.338096] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2751.460833] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2756.580467] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2757.697687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2758.213410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2762.887062] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 2770.528779] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2775.260958] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2780.826945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2781.113941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2784.983652] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 2791.356752] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2791.555151] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2799.585874] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2799.595359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2799.605375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2801.639361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2801.650495] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2801.666949] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2805.209385] Lustre: server umount lustre-OST0000 complete [ 2809.312123] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2809.325159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2809.363361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2809.367538] Lustre: Skipped 1 previous similar message [ 2811.873685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2811.875532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2811.891751] Lustre: Skipped 1 previous similar message [ 2816.993079] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2817.014931] LustreError: Skipped 1 previous similar message [ 2817.588902] Lustre: server umount lustre-MDT0000 complete [ 2821.405144] LustreError: 103578:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202283 with bad export cookie 3024547532370111819 [ 2821.410417] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2821.419238] LustreError: 103578:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2821.678981] Lustre: server umount lustre-MDT0001 complete [ 2834.097446] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 2836.692866] Key type lgssc unregistered [ 2836.926688] LNet: 107253:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2836.932736] LNet: Removed LNI 192.168.202.119@tcp [ 2837.528729] Key type .llcrypt unregistered [ 2837.531919] Key type ._llcrypt unregistered [ 2851.635573] alg: No test for adler32 (adler32-zlib) [ 2852.392640] Key type ._llcrypt registered [ 2852.395836] Key type .llcrypt registered [ 2852.521887] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2853.675657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2854.077876] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 2854.318709] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 2854.321408] LNet: Accept secure, port 988 [ 2855.991165] Key type lgssc registered [ 2857.170462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2863.060639] LustreError: 108378:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 2863.063468] LustreError: 108382:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 2863.072128] LustreError: 108378:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4991 [ 2863.076795] LustreError: 108378:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2863.093837] LustreError: 108378:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2863.102624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2864.650548] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2864.726553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2869.747494] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2873.607581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2874.011782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2878.079437] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 2883.249086] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing lsmod [ 2886.649967] LustreError: 109690:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 2886.683522] LustreError: 109691:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 2886.688788] LustreError: 109690:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4975 [ 2886.694280] LustreError: 109690:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2886.707271] LustreError: 109690:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2886.712475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2887.252794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2894.823964] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2894.825085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2894.827112] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2894.831784] Lustre: Skipped 2 previous similar messages [ 2894.863344] Lustre: Skipped 2 previous similar messages [ 2896.089965] Lustre: server umount lustre-MDT0000 complete [ 2899.954417] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2899.977188] LustreError: Skipped 2 previous similar messages [ 2900.459132] LustreError: 109730:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202362 with bad export cookie 18351381222906195951 [ 2900.460276] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2900.472701] LustreError: 109730:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2900.803745] Lustre: server umount lustre-MDT0001 complete [ 2910.691371] Lustre: 110285:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202366/real 1776202366] req@00000000bd6efc08 x1862483119712448/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776202372 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2910.868191] Lustre: server umount lustre-OST0000 complete [ 2919.976263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2920.444740] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2920.517569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2924.539925] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 2925.550183] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2934.120764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2934.380556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2940.129206] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 2948.661360] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2955.719096] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2962.616723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2962.967946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2967.952977] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 2970.158549] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 2975.554700] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2981.863766] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2981.874375] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2981.896612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2984.026394] Lustre: server umount lustre-OST0000 complete [ 2990.562593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2990.564697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2990.584186] Lustre: Skipped 1 previous similar message [ 2990.594376] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2991.391175] Lustre: 107783:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202446/real 1776202446] req@00000000f5929183 x1862483119726784/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1776202453 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 2992.415302] Lustre: 107783:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202447/real 1776202447] req@00000000f9c7baa7 x1862483119727104/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776202454 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2995.712315] Lustre: server umount lustre-MDT0000 complete [ 2996.707492] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2996.720843] LustreError: Skipped 2 previous similar messages [ 2997.729602] Lustre: 107782:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202452/real 1776202452] req@000000004082f976 x1862483119728000/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776202459 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3000.185511] LustreError: 110741:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202462 with bad export cookie 18351381222906196665 [ 3000.187703] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3000.193298] LustreError: 110741:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3012.822505] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 3016.116539] Key type lgssc unregistered [ 3016.389148] LNet: 114423:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3016.400703] LNet: Removed LNI 192.168.202.119@tcp [ 3017.198359] Key type .llcrypt unregistered [ 3017.204305] Key type ._llcrypt unregistered [ 3041.084625] alg: No test for adler32 (adler32-zlib) [ 3041.841815] Key type ._llcrypt registered [ 3041.847426] Key type .llcrypt registered [ 3041.976270] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3042.807834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3043.361522] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 3043.524701] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 3043.528668] LNet: Accept secure, port 988 [ 3045.159226] Key type lgssc registered [ 3046.155810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3052.868866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3054.309964] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3054.365102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3058.053687] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 3059.681107] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3064.802660] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3066.030666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3066.413384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3070.618802] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 3077.447487] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3082.746667] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3088.614972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3089.054577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3093.902953] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 3100.377805] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3104.434760] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 3109.992991] Lustre: Setting parameter lustre-client.llite.some_wrong_param in log lustre-client [ 3113.510570] Lustre: Modifying parameter lustre-client.llite.some_wrong_param in log lustre-client [ 3120.101877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3120.135574] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3120.135953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3120.166990] Lustre: Skipped 1 previous similar message [ 3123.851055] Lustre: server umount lustre-OST0000 complete [ 3130.347867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3130.352944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3130.360175] Lustre: Skipped 2 previous similar messages [ 3130.371922] Lustre: Skipped 1 previous similar message [ 3134.568328] Lustre: server umount lustre-MDT0000 complete [ 3135.456101] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3135.480476] LustreError: Skipped 1 previous similar message [ 3139.856281] LustreError: 115866:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202601 with bad export cookie 13133142670904610590 [ 3139.858905] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3139.880052] LustreError: 115866:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3140.399976] Lustre: server umount lustre-MDT0001 complete [ 3156.229829] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 3159.871938] Key type lgssc unregistered [ 3160.264700] LNet: 119644:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3160.270492] LNet: Removed LNI 192.168.202.119@tcp [ 3161.231498] Key type .llcrypt unregistered [ 3161.239602] Key type ._llcrypt unregistered [ 3177.785653] alg: No test for adler32 (adler32-zlib) [ 3178.541412] Key type ._llcrypt registered [ 3178.544477] Key type .llcrypt registered [ 3178.657132] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3180.034559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3180.835766] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 3181.096623] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 3181.102595] LNet: Accept secure, port 988 [ 3182.783127] Key type lgssc registered [ 3183.825422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3192.034389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3193.494237] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3193.562704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3198.763559] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 3198.950133] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3204.072425] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3207.823369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3208.107374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3211.913585] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 3218.384725] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3224.008961] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3230.800759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3231.199440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3235.515393] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 3244.355238] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3244.536683] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 3253.728631] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3253.736550] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3253.766545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3254.757156] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3254.760264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3254.764856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3254.779826] Lustre: Skipped 1 previous similar message [ 3258.893065] Lustre: server umount lustre-OST0000 complete [ 3264.480401] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3264.493323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3264.505637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3269.521275] Lustre: server umount lustre-MDT0000 complete [ 3270.116273] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3270.149091] LustreError: Skipped 1 previous similar message [ 3274.360328] LustreError: 120906:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202736 with bad export cookie 9942100016123129599 [ 3274.362616] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3274.371245] LustreError: 120906:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3275.069402] Lustre: server umount lustre-MDT0001 complete [ 3286.945959] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 3289.257052] Key type lgssc unregistered [ 3289.561763] LNet: 124582:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3289.571147] LNet: Removed LNI 192.168.202.119@tcp [ 3290.325923] Key type .llcrypt unregistered [ 3290.328640] Key type ._llcrypt unregistered [ 3307.997499] alg: No test for adler32 (adler32-zlib) [ 3308.757877] Key type ._llcrypt registered [ 3308.763537] Key type .llcrypt registered [ 3308.854967] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3309.712257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3309.994460] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 3310.142167] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 3310.149206] LNet: Accept secure, port 988 [ 3311.831326] Key type lgssc registered [ 3312.826297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3320.139049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3321.558556] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3321.589382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3326.738795] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 3326.947655] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3332.088813] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3337.196197] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3338.514373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3339.088552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3344.777190] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 3354.067644] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3361.910529] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3368.524676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3368.812638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3372.942815] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 3373.099608] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 3380.262551] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3387.057345] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 3391.202920] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 3391.211744] Lustre: Skipped 2 previous similar messages [ 3395.701533] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 3395.705392] Lustre: Skipped 2 previous similar messages [ 3396.180584] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3396.261486] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3404.066583] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 3404.072133] Lustre: Skipped 2 previous similar messages [ 3405.195145] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.19@tcp 192.168.0.[2,10]@tcp [ 3416.034206] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3416.045241] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3416.056390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3417.064192] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3417.070993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3417.078833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3417.223890] Lustre: server umount lustre-OST0000 complete [ 3421.152125] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3421.161376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3421.183505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3424.224486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3424.226997] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3427.201431] Lustre: server umount lustre-MDT0000 complete [ 3429.350532] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3429.379113] LustreError: Skipped 1 previous similar message [ 3430.924847] LustreError: 126031:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202893 with bad export cookie 3384273924217884358 [ 3430.927345] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3430.935429] LustreError: 126031:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3431.221187] Lustre: server umount lustre-MDT0001 complete [ 3442.889714] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 3445.838777] Key type lgssc unregistered [ 3446.083278] LNet: 130566:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3446.089462] LNet: Removed LNI 192.168.202.119@tcp [ 3447.125741] Key type .llcrypt unregistered [ 3447.131643] Key type ._llcrypt unregistered [ 3478.476585] alg: No test for adler32 (adler32-zlib) [ 3479.231069] Key type ._llcrypt registered [ 3479.233105] Key type .llcrypt registered [ 3479.406924] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3480.570353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3481.306116] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 3481.557895] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 3481.560570] LNet: Accept secure, port 988 [ 3483.223152] Key type lgssc registered [ 3484.269594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3493.097183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3494.539792] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3494.580682] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3494.611409] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.202.19@tcp 192.168.0.[2,10]@tcp [ 3494.631729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3498.845098] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 3500.021560] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3505.127677] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3507.529683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3507.768607] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3507.820666] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.19@tcp 192.168.0.[2,10]@tcp [ 3507.870046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3512.200188] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 3519.394464] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3525.357421] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3531.402465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3531.813065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3535.542801] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 3542.168251] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3545.153264] Lustre: lustre-OST0000: deleting orphan objects from 0x0:167 to 0x0:193 [ 3560.430933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3560.432426] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3560.437813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3560.448583] Lustre: Skipped 1 previous similar message [ 3562.827117] Lustre: server umount lustre-OST0000 complete [ 3569.120271] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3569.132391] LustreError: Skipped 1 previous similar message [ 3569.151280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3569.172815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3569.178322] Lustre: Skipped 1 previous similar message [ 3570.658762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3570.662418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3572.971759] Lustre: server umount lustre-MDT0000 complete [ 3575.777939] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3575.791741] LustreError: Skipped 1 previous similar message [ 3578.106753] LustreError: 132192:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203040 with bad export cookie 2498723684655089429 [ 3578.117846] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3578.616992] Lustre: server umount lustre-MDT0001 complete [ 3593.970106] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 3597.311369] Key type lgssc unregistered [ 3597.587071] LNet: 135928:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3597.591587] LNet: Removed LNI 192.168.202.119@tcp [ 3598.279495] Key type .llcrypt unregistered [ 3598.281389] Key type ._llcrypt unregistered [ 3626.661726] alg: No test for adler32 (adler32-zlib) [ 3627.419605] Key type ._llcrypt registered [ 3627.431208] Key type .llcrypt registered [ 3627.589862] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 3639.252816] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3640.565936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3640.981909] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 3641.185683] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 3641.190719] LNet: Accept secure, port 988 [ 3642.863175] Key type lgssc registered [ 3644.008395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3652.241681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3658.983099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3664.442208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3670.041119] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3676.489288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3676.555936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3677.806712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3677.836858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3677.874836] Lustre: lustre-MDT0000: new disk, initializing [ 3677.968603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3677.987944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3682.321965] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 3693.432171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3693.530475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3693.615198] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3693.718940] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3693.730564] Lustre: Skipped 1 previous similar message [ 3693.877332] Lustre: lustre-MDT0001: new disk, initializing [ 3693.950574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3693.997973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3694.012292] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3699.358771] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 3709.555770] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3715.022327] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3721.589657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3721.661289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3721.667330] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3721.860808] Lustre: lustre-OST0000: new disk, initializing [ 3721.864682] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3721.937601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3725.709795] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 3738.090627] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3744.141302] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3744.343440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3748.835989] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3749.091713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3751.393667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3751.401508] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3751.417615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3756.515121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3756.527429] Lustre: Skipped 1 previous similar message [ 3757.396472] Lustre: server umount lustre-OST0000 complete [ 3765.734861] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3765.742391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3765.759818] Lustre: Skipped 1 previous similar message [ 3765.764804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3765.773802] Lustre: Skipped 1 previous similar message [ 3766.756758] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3767.957336] Lustre: server umount lustre-MDT0000 complete [ 3771.142163] LustreError: 139445:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203233 with bad export cookie 6115181636605399024 [ 3771.147941] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3771.158209] LustreError: 139445:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3771.451269] Lustre: server umount lustre-MDT0001 complete [ 3779.383056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3779.752836] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3779.829820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3783.915464] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 3785.185913] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3790.308956] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3790.887707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3791.134030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3794.733914] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 3802.116733] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3807.969346] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3814.772841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3815.046540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3819.997620] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 3821.239538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3821.256232] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3828.955875] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3835.225139] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3835.413260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3841.245914] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3841.462818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3849.510542] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3849.567749] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3849.576182] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3849.693100] Lustre: lustre-OST0001: new disk, initializing [ 3849.701262] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3849.792561] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3853.438802] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 3856.970466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3856.977870] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3860.263554] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3860.408771] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3864.338824] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 3864.507276] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3869.454342] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8fc544a0c000.ost_server_uuid 40 [ 3870.703516] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8fc544a0c000.ost_server_uuid in FULL state after 0 sec [ 3882.977144] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3882.996783] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3883.004033] LustreError: Skipped 1 previous similar message [ 3883.008690] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3883.023241] Lustre: Skipped 1 previous similar message [ 3883.041780] Lustre: Skipped 3 previous similar messages [ 3884.145968] Lustre: server umount lustre-OST0001 complete [ 3888.104314] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3888.110554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3888.114057] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3888.115462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3888.139247] LustreError: Skipped 1 previous similar message [ 3888.143887] Lustre: Skipped 1 previous similar message [ 3893.238098] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3893.252538] LustreError: Skipped 1 previous similar message [ 3893.908661] Lustre: server umount lustre-OST0000 complete [ 3898.337320] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3898.339281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3898.358370] Lustre: Skipped 1 previous similar message [ 3898.374553] Lustre: Skipped 4 previous similar messages [ 3904.041833] Lustre: server umount lustre-MDT0000 complete [ 3907.320160] LustreError: 143791:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203369 with bad export cookie 6115181636605399878 [ 3907.334000] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3907.337421] LustreError: 143791:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3927.482509] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 3930.327660] Key type lgssc unregistered [ 3930.605321] LNet: 150464:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3930.621783] LNet: Removed LNI 192.168.202.119@tcp [ 3931.722353] Key type .llcrypt unregistered [ 3931.726260] Key type ._llcrypt unregistered [ 3965.150224] alg: No test for adler32 (adler32-zlib) [ 3965.908401] Key type ._llcrypt registered [ 3965.910370] Key type .llcrypt registered [ 3966.025284] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 3981.174917] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3982.730931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3983.416852] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 3983.631568] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 3983.638922] LNet: Accept secure, port 988 [ 3985.367721] Key type lgssc registered [ 3986.712807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3997.811052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4005.538682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4012.600994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4018.824326] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4025.654082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4025.762584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4027.079116] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4027.118165] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4027.205483] Lustre: lustre-MDT0000: new disk, initializing [ 4027.328503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4027.350526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4032.455520] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 4043.515299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4043.631260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4043.753968] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4043.784991] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4043.789115] Lustre: Skipped 1 previous similar message [ 4043.858814] Lustre: lustre-MDT0001: new disk, initializing [ 4043.953160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4044.027186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4044.043954] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4048.427216] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 4056.980235] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4061.813686] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4067.325759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4067.404781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4067.410962] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4067.564992] Lustre: lustre-OST0000: new disk, initializing [ 4067.567947] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4067.630585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4072.006167] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 4074.570390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4074.580140] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4081.444911] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4086.509651] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4086.761884] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4091.563729] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4091.737578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4095.458875] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4095.461161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4095.478135] LustreError: Skipped 1 previous similar message [ 4095.478887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4095.500825] Lustre: Skipped 1 previous similar message [ 4099.822407] Lustre: server umount lustre-OST0000 complete [ 4105.697836] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4105.700340] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4105.710301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4105.717718] Lustre: Skipped 2 previous similar messages [ 4109.420718] Lustre: server umount lustre-MDT0000 complete [ 4110.817140] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4110.842796] LustreError: Skipped 1 previous similar message [ 4111.843527] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4111.875554] LustreError: Skipped 1 previous similar message [ 4113.268980] LustreError: 154576:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203575 with bad export cookie 10989770179563088617 [ 4113.274527] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4113.286753] LustreError: 154576:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4113.497833] Lustre: server umount lustre-MDT0001 complete [ 4120.759930] Lustre: DEBUG MARKER: == conf-sanity test 47: server restart does not make client loss lru_resize settings ========================================================== 17:53:02 (1776203582) [ 4137.811411] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 4146.643748] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 4156.372310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4162.021737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4167.697199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4172.979975] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4179.607581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4179.677152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4179.869085] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4179.899688] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4179.976884] Lustre: lustre-MDT0000: new disk, initializing [ 4180.055154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4180.070730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4183.916882] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 4195.814879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4195.900880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4195.999715] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4196.069185] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4196.075146] Lustre: Skipped 1 previous similar message [ 4196.246225] Lustre: lustre-MDT0001: new disk, initializing [ 4196.351204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4196.395835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4196.417067] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4201.667440] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 4213.764848] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4220.254679] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4228.457331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4228.519404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4228.526784] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4228.884887] Lustre: lustre-OST0000: new disk, initializing [ 4228.892061] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4229.035732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4234.054705] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 4237.415164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4237.432656] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4243.066448] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4246.504347] Lustre: Failing over lustre-OST0000 [ 4246.611769] Lustre: server umount lustre-OST0000 complete [ 4255.199358] Lustre: 152805:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776203710/real 1776203710] req@000000007da1c1d4 x1862484303578688/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1776203717 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 4255.222196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4255.244924] Lustre: Skipped 1 previous similar message [ 4256.223223] Lustre: 152803:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776203711/real 1776203711] req@00000000b612b9df x1862484303579136/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1776203718 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 4260.338312] Lustre: 152805:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776203715/real 1776203715] req@0000000014305924 x1862484303579712/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776203722 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4260.382890] Lustre: 152805:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4262.368342] Lustre: 152802:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776203717/real 1776203717] req@000000009e70843f x1862484303580096/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776203724 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4265.495330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4265.893307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4265.916271] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4269.787378] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 4272.035662] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4273.205392] Lustre: Failing over lustre-MDT0000 [ 4273.241646] LustreError: 152801:0:(import.c:692:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4273.315497] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 4273.414848] Lustre: server umount lustre-MDT0000 complete [ 4274.386377] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4274.435387] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4274.437981] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.119@tcp (at 0@lo) [ 4277.096381] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4277.097932] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4277.126764] Lustre: Skipped 3 previous similar messages [ 4277.149103] LustreError: Skipped 1 previous similar message [ 4281.516674] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4281.534560] LustreError: Skipped 2 previous similar messages [ 4284.385707] Lustre: 152804:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776203739/real 1776203739] req@00000000b3a6560c x1862484303586688/t0(0) o400->MGC192.168.202.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776203746 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4284.401669] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4290.532685] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4290.554284] LustreError: Skipped 13 previous similar messages [ 4292.934271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4301.800357] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0x988381c08409ee3f to 0x988381c08409f30f [ 4301.841283] Lustre: MGC192.168.202.119@tcp: Connection restored to 192.168.202.119@tcp (at 0@lo) [ 4302.129471] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4302.216236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4302.295889] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4302.929101] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4307.464085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 4307.492362] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4307.561405] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 4308.325816] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 4317.666509] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4317.672397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4317.682081] Lustre: Skipped 1 previous similar message [ 4322.793777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4322.808853] Lustre: Skipped 2 previous similar messages [ 4323.297107] Lustre: server umount lustre-OST0000 complete [ 4327.914156] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4327.915450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4327.949035] Lustre: Skipped 1 previous similar message [ 4333.025354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4333.036433] Lustre: Skipped 3 previous similar messages [ 4341.215230] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4341.450636] Lustre: server umount lustre-MDT0000 complete [ 4343.264224] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4343.279720] LustreError: Skipped 13 previous similar messages [ 4344.935120] LustreError: 162436:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203807 with bad export cookie 10989770179563090703 [ 4344.947234] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4356.601273] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 4359.727396] Key type lgssc unregistered [ 4359.976667] LNet: 167437:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4359.982589] LNet: Removed LNI 192.168.202.119@tcp [ 4360.645550] Key type .llcrypt unregistered [ 4360.647950] Key type ._llcrypt unregistered [ 4383.030460] alg: No test for adler32 (adler32-zlib) [ 4383.782573] Key type ._llcrypt registered [ 4383.784405] Key type .llcrypt registered [ 4383.902223] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 4384.861529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4385.413840] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 4385.589580] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 4385.592989] LNet: Accept secure, port 988 [ 4387.239121] Key type lgssc registered [ 4388.258880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4395.376478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4396.919825] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4396.988715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4400.825219] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 4402.144195] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4407.267267] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4408.337939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4408.697982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4412.865819] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 4420.340293] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4426.210925] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4433.516357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4433.991140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4438.620653] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 4445.922983] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4447.343859] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 4502.722340] Lustre: lustre-MDT0000: Client 653ccca4-8c93-4ed5-94c2-23441c8b6148 (at 192.168.202.19@tcp) reconnecting [ 4531.404322] Lustre: lustre-MDT0000: Client 653ccca4-8c93-4ed5-94c2-23441c8b6148 (at 192.168.202.19@tcp) reconnecting [ 4546.820374] Lustre: lustre-MDT0000: Client 653ccca4-8c93-4ed5-94c2-23441c8b6148 (at 192.168.202.19@tcp) reconnecting [ 4566.731284] Lustre: lustre-MDT0000: Client 653ccca4-8c93-4ed5-94c2-23441c8b6148 (at 192.168.202.19@tcp) reconnecting [ 4581.574251] Lustre: lustre-MDT0000: Client 653ccca4-8c93-4ed5-94c2-23441c8b6148 (at 192.168.202.19@tcp) reconnecting [ 4597.940732] Lustre: lustre-MDT0000: Client 653ccca4-8c93-4ed5-94c2-23441c8b6148 (at 192.168.202.19@tcp) reconnecting [ 4722.385762] Lustre: lustre-MDT0000: Client 653ccca4-8c93-4ed5-94c2-23441c8b6148 (at 192.168.202.19@tcp) reconnecting [ 4812.481541] Lustre: lustre-MDT0000: Client 653ccca4-8c93-4ed5-94c2-23441c8b6148 (at 192.168.202.19@tcp) reconnecting [ 4923.098780] Lustre: lustre-MDT0000: Client 653ccca4-8c93-4ed5-94c2-23441c8b6148 (at 192.168.202.19@tcp) reconnecting [ 4923.106467] Lustre: Skipped 1 previous similar message [ 4941.882849] Lustre: Failing over lustre-MDT0000 [ 4942.049350] Lustre: server umount lustre-MDT0000 complete [ 4942.306219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4942.319189] Lustre: Skipped 2 previous similar messages [ 4942.320387] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4942.331672] LustreError: Skipped 2 previous similar messages [ 4947.428158] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4947.443094] LustreError: Skipped 3 previous similar messages [ 4949.471563] Lustre: 168210:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776204404/real 1776204404] req@00000000018ef711 x1862484725451328/t0(0) o400->MGC192.168.202.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776204411 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4949.501483] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4955.621908] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4955.644602] LustreError: Skipped 10 previous similar messages [ 4961.750837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4966.889786] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0x5ad43ee7ac6b9f78 to 0x5ad43ee7ac6e0b99 [ 4966.921648] Lustre: MGC192.168.202.119@tcp: Connection restored to 192.168.202.119@tcp (at 0@lo) [ 4967.049116] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (not set up) [ 4967.243505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4967.304605] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4968.079199] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4971.862271] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 4972.540910] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 4972.571473] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4972.659724] Lustre: lustre-OST0000: deleting orphan objects from 0x0:68 to 0x0:97 [ 4981.995353] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4983.696684] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5003.234153] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5003.238354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5003.245814] Lustre: Skipped 1 previous similar message [ 5003.258273] Lustre: Skipped 2 previous similar messages [ 5006.949967] Lustre: server umount lustre-MDT0000 complete [ 5008.358684] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5008.392370] LustreError: Skipped 14 previous similar messages [ 5010.353016] LustreError: 171513:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204472 with bad export cookie 6544925323227433881 [ 5010.353484] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5010.364707] LustreError: 171513:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5013.475543] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5013.502490] Lustre: Skipped 1 previous similar message [ 5013.506363] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5015.520642] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5016.730744] Lustre: server umount lustre-MDT0001 complete [ 5034.975291] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5035.171506] Lustre: server umount lustre-OST0000 complete [ 5044.385432] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 5053.853637] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 5063.164338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5069.151851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5077.354808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5084.655785] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5091.611342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5091.685772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5091.933353] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5091.988689] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5092.064815] Lustre: lustre-MDT0000: new disk, initializing [ 5092.129715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5092.143647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5096.594959] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 5106.155841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5106.233555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5106.324874] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5106.363426] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5106.367461] Lustre: Skipped 1 previous similar message [ 5106.442514] Lustre: lustre-MDT0001: new disk, initializing [ 5106.523490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5106.558174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5106.571053] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5109.976901] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 5118.196924] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5123.533573] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5130.169695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5130.216875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5130.226724] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5130.472675] Lustre: lustre-OST0000: new disk, initializing [ 5130.477886] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5130.565139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5135.273188] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 5139.511209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5139.520101] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5144.247166] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5148.582349] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5148.837239] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5152.801862] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5153.026578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5155.296250] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5155.303237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5155.318231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5155.323443] Lustre: Skipped 2 previous similar messages [ 5160.676588] Lustre: server umount lustre-OST0000 complete [ 5165.543924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5165.563165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5165.565697] Lustre: Skipped 2 previous similar messages [ 5165.584066] Lustre: Skipped 3 previous similar messages [ 5170.552374] Lustre: server umount lustre-MDT0000 complete [ 5170.661838] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5170.671219] LustreError: Skipped 4 previous similar messages [ 5174.025280] LustreError: 175837:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204636 with bad export cookie 6544925323227434602 [ 5174.028467] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5174.039742] LustreError: 175837:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5181.053979] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:10:42 (1776204642) [ 5195.785422] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 5204.380523] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 5213.995345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5220.117659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5225.920992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5231.138944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5237.141767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5237.197895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5237.354840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5237.375463] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5237.457351] Lustre: lustre-MDT0000: new disk, initializing [ 5237.543848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5237.562292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5241.490792] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 5251.595120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5251.672556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5251.801385] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5251.840694] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5251.844809] Lustre: Skipped 1 previous similar message [ 5251.969432] Lustre: lustre-MDT0001: new disk, initializing [ 5252.158660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5252.172620] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5257.369221] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 5267.750609] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5274.903404] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5283.149587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5283.277510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5283.303472] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5283.703777] Lustre: lustre-OST0000: new disk, initializing [ 5283.709123] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5289.550499] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 5292.575520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5292.589136] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5298.027927] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5307.875227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5307.909358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5307.912251] Lustre: Skipped 1 previous similar message [ 5310.728168] Lustre: server umount lustre-OST0000 complete [ 5310.734500] Lustre: Skipped 1 previous similar message [ 5328.351800] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5329.376931] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5329.398126] LustreError: Skipped 1 previous similar message [ 5331.963680] LustreError: 182819:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204794 with bad export cookie 6544925323227435456 [ 5331.965762] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5331.971113] LustreError: 182819:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5340.018398] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:13:21 (1776204801) [ 5356.184971] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 5365.914556] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 5375.629415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5382.746206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5389.517961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5397.350245] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5406.223384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5406.326735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5406.627481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5406.673976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5406.722828] Lustre: lustre-MDT0000: new disk, initializing [ 5406.830186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5406.853556] Lustre: Skipped 2 previous similar messages [ 5406.885741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5411.601284] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 5421.380944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5421.499076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5421.716600] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5422.067476] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5425.705677] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 5433.158889] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5437.931333] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5443.312798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5443.367513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5443.378349] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5443.623901] Lustre: lustre-OST0000: new disk, initializing [ 5443.627266] Lustre: Skipped 1 previous similar message [ 5443.631285] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5443.647249] Lustre: Skipped 2 previous similar messages [ 5447.539994] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 5450.845788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5450.857299] Lustre: Skipped 1 previous similar message [ 5450.868621] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5455.769743] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5461.477451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5461.482682] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5461.485273] Lustre: Skipped 3 previous similar messages [ 5461.487070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5461.487083] Lustre: Skipped 6 previous similar messages [ 5461.498517] LustreError: Skipped 1 previous similar message [ 5467.136196] Lustre: server umount lustre-OST0000 complete [ 5467.140061] Lustre: Skipped 2 previous similar messages [ 5476.834860] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5476.848014] LustreError: Skipped 1 previous similar message [ 5479.352802] LustreError: 189289:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204941 with bad export cookie 6544925323227436541 [ 5479.360968] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5479.365973] LustreError: 189289:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5490.237162] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 5493.110547] Key type lgssc unregistered [ 5493.393849] LNet: 193252:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5493.408022] LNet: Removed LNI 192.168.202.119@tcp [ 5494.208638] Key type .llcrypt unregistered [ 5494.211506] Key type ._llcrypt unregistered [ 5518.340736] alg: No test for adler32 (adler32-zlib) [ 5519.100465] Key type ._llcrypt registered [ 5519.103056] Key type .llcrypt registered [ 5519.214389] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 5520.733812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5521.571995] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 5521.867197] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 5521.871490] LNet: Accept secure, port 988 [ 5523.559116] Key type lgssc registered [ 5524.594409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5532.320631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5533.910249] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5533.980474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5538.423894] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 5539.296989] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5544.421744] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5546.431836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5546.789260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5550.675952] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 5557.802470] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5562.792913] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5568.021147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5568.353700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5571.924927] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 5580.067930] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5584.815833] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 5603.298801] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5603.315845] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5603.328885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5605.344271] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5605.353242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5605.357068] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5605.381291] Lustre: Skipped 1 previous similar message [ 5607.753983] Lustre: server umount lustre-OST0000 complete [ 5613.535841] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5613.540289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5613.549109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5615.586608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5615.586926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5616.755563] Lustre: server umount lustre-MDT0000 complete [ 5619.649302] LustreError: 194716:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205081 with bad export cookie 15886829011647915867 [ 5619.651402] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5619.662396] LustreError: 194716:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5619.871179] Lustre: server umount lustre-MDT0001 complete [ 5630.288370] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 5632.389785] Key type lgssc unregistered [ 5632.626436] LNet: 198416:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5632.633433] LNet: Removed LNI 192.168.202.119@tcp [ 5633.396569] Key type .llcrypt unregistered [ 5633.400382] Key type ._llcrypt unregistered [ 5654.032704] alg: No test for adler32 (adler32-zlib) [ 5654.786475] Key type ._llcrypt registered [ 5654.791301] Key type .llcrypt registered [ 5654.882979] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 5655.603123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5656.061687] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 5656.236888] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 5656.238971] LNet: Accept secure, port 988 [ 5657.863739] Key type lgssc registered [ 5658.677521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5664.068587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5665.387643] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5665.426624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5668.535931] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 5670.898687] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5675.521965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5675.801319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5679.186522] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 5685.191795] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5690.314869] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5695.552938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5695.868205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5700.196682] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 5709.577763] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5715.325287] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 5720.548958] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5720.551111] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5720.562787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5720.568082] Lustre: Skipped 1 previous similar message [ 5721.294548] Lustre: lustre-OST0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 5721.303615] Lustre: Skipped 1 previous similar message [ 5725.665515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5730.786611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5730.799749] Lustre: Skipped 3 previous similar messages [ 5734.367179] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5734.516362] Lustre: server umount lustre-OST0000 complete [ 5742.843122] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8fc5481bb800.ost_server_uuid 40 [ 5744.753392] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5762.530418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5762.545952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5762.556449] Lustre: Skipped 2 previous similar messages [ 5764.842344] Lustre: server umount lustre-MDT0000 complete [ 5767.655387] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5767.667114] LustreError: Skipped 1 previous similar message [ 5768.137446] LustreError: 202896:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205230 with bad export cookie 14787142397137500606 [ 5768.141903] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5768.152402] LustreError: 202896:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5768.443749] Lustre: server umount lustre-MDT0001 complete [ 5776.772526] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 18:20:38 (1776205238) [ 5783.448760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5783.828842] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5783.901410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5788.340528] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 5789.156169] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5796.192388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5796.545610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5800.747136] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 5807.941834] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5813.490500] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5820.441955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5820.814729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5825.690501] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 5833.013759] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5834.168353] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 5839.396696] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5839.508871] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5839.525137] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 5839.646638] Lustre: lustre-OST0001: new disk, initializing [ 5839.651094] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5839.722467] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5844.122701] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 5848.124127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5848.135443] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5853.092322] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5856.737234] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5856.750075] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5856.793654] Lustre: Skipped 1 previous similar message [ 5856.799959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5858.806101] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5862.086628] Lustre: server umount lustre-OST0000 complete [ 5863.914121] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5863.926698] LustreError: Skipped 2 previous similar messages [ 5869.135788] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5869.351234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5873.393767] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5873.592134] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5884.387616] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5884.403860] LustreError: Skipped 11 previous similar messages [ 5889.549049] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5889.555598] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5889.564821] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5889.565710] Lustre: Skipped 1 previous similar message [ 5889.589784] Lustre: Skipped 4 previous similar messages [ 5894.179491] Lustre: server umount lustre-OST0001 complete [ 5899.236930] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5899.257068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5904.166805] Lustre: server umount lustre-MDT0000 complete [ 5907.881170] LustreError: 203851:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205369 with bad export cookie 14787142397137501670 [ 5907.884735] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5907.891678] LustreError: 203851:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5913.311769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5917.318878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5921.345226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5925.048605] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5931.604734] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 18:23:13 (1776205393) [ 5938.262875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5938.630712] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5938.709771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5938.901855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5943.826933] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 5952.827429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5952.890668] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5952.920485] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5957.399182] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 5964.845386] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5970.585902] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5976.929286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5977.098589] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5981.637211] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 5987.385587] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 5988.438442] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5994.614423] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5994.699585] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5999.282744] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 6006.176382] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6012.065964] Lustre: lustre-OST0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 6012.084218] Lustre: Skipped 2 previous similar messages [ 6012.392917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6012.402936] Lustre: Skipped 2 previous similar messages [ 6014.587937] Lustre: server umount lustre-OST0000 complete [ 6014.591897] Lustre: Skipped 1 previous similar message [ 6017.174434] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6017.196450] LustreError: Skipped 7 previous similar messages [ 6032.863896] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6048.324275] LustreError: 213249:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205510 with bad export cookie 14787142397137503056 [ 6048.336563] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6048.337660] LustreError: 213249:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6048.604760] Lustre: server umount lustre-MDT0001 complete [ 6048.607226] Lustre: Skipped 2 previous similar messages [ 6055.341621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6059.966622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6064.401306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6068.825303] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6076.184726] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 18:25:37 (1776205537) [ 6093.474283] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 6102.376386] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 6110.875622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6116.164661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6121.329979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6126.393641] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6131.828969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6131.876922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6132.077443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6132.165858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6132.263185] Lustre: lustre-MDT0000: new disk, initializing [ 6132.335986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6132.340272] Lustre: Skipped 3 previous similar messages [ 6132.351769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6136.808445] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 6146.984786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6147.040298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6147.101050] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6147.123551] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6147.127148] Lustre: Skipped 1 previous similar message [ 6147.202715] Lustre: lustre-MDT0001: new disk, initializing [ 6147.334157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6147.353866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6151.291151] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 6159.875103] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6165.816500] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6172.854215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6172.948629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6172.953479] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6173.233525] Lustre: lustre-OST0000: new disk, initializing [ 6173.241777] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6178.703400] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 6178.912413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6178.915302] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6188.954513] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6194.778842] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6195.041794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6199.585284] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6199.767104] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6204.902873] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6204.906617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6204.921106] Lustre: Skipped 5 previous similar messages [ 6204.929315] Lustre: Skipped 7 previous similar messages [ 6207.649832] Lustre: server umount lustre-OST0000 complete [ 6214.113722] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6214.126721] LustreError: Skipped 1 previous similar message [ 6220.262650] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6220.287705] LustreError: Skipped 12 previous similar messages [ 6221.374617] LustreError: 218521:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205683 with bad export cookie 14787142397137504568 [ 6221.379870] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6221.388879] LustreError: 218521:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6230.234701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6234.674228] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 6242.013670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6245.808495] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 6252.285897] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6257.685725] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6264.696447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6265.009891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6265.015215] Lustre: Skipped 4 previous similar messages [ 6269.345367] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 6276.818102] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6281.692930] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6281.952711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6287.691344] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6287.993988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6291.937626] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6291.939875] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6291.985386] Lustre: Skipped 2 previous similar messages [ 6305.072854] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6311.585284] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6340.607589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6345.132154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6345.155389] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6345.163833] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 6345.168838] Lustre: lustre-OST0000: deleting orphan objects from 0x0:34 to 0x0:65 [ 6345.187483] Lustre: Skipped 1 previous similar message [ 6345.193170] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 6346.331875] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 6354.757834] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6356.673344] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6365.666773] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6365.673080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6365.687798] LustreError: Skipped 1 previous similar message [ 6365.710177] Lustre: Skipped 7 previous similar messages [ 6367.725121] Lustre: server umount lustre-OST0000 complete [ 6367.731523] Lustre: Skipped 3 previous similar messages [ 6380.902193] LustreError: 222868:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205843 with bad export cookie 14787142397137505422 [ 6380.906670] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6380.910240] LustreError: 222868:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6389.530570] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 18:30:51 (1776205851) [ 6395.556912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6395.991285] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6396.001782] LustreError: Skipped 5 previous similar messages [ 6400.638389] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 6407.439165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6411.359758] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 6417.952271] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6422.309767] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6428.612921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6432.940540] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 6439.384552] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6443.455890] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6446.870929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 6450.889345] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6451.084538] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6457.224168] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6457.298749] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6457.303852] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 6457.410425] Lustre: lustre-OST0001: new disk, initializing [ 6457.415872] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 6461.129734] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 6465.122376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6465.129618] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6469.387927] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6473.457892] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 6473.679777] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6477.640873] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 6477.831749] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6480.869802] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6480.872315] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6480.888247] LustreError: Skipped 2 previous similar messages [ 6480.894707] Lustre: Skipped 5 previous similar messages [ 6491.021659] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 6491.175596] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6495.576892] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 6495.855532] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6524.730596] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6524.878483] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6524.893075] Lustre: Skipped 5 previous similar messages [ 6530.025540] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6530.041721] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6530.054209] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to (at 0@lo) [ 6530.064171] Lustre: Skipped 1 previous similar message [ 6530.444243] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 6538.198271] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6540.044863] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6579.167973] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6592.796732] LustreError: 229060:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206054 with bad export cookie 14787142397137506836 [ 6592.800412] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6592.810193] LustreError: 229060:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6607.327139] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6613.382143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6616.677198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6619.794300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6622.956083] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6628.328423] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 18:34:49 (1776206089) [ 6633.696501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6633.986913] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6634.015576] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6638.243162] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 6645.367727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6645.453401] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6645.486508] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6649.497089] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 6655.448582] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6660.818990] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6666.260214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6666.397213] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6670.171697] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 6677.059295] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6685.783682] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6685.834487] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6690.280560] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 6696.945256] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6698.592671] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6702.531268] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 6702.729400] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6707.469412] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 6707.665929] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6712.259353] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8fc542e0f000.ost_server_uuid 40 [ 6713.474426] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8fc542e0f000.ost_server_uuid in IDLE state after 0 sec [ 6714.707556] Lustre: Permanently deactivating lustre-OST0001 [ 6714.727082] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 6717.960874] Lustre: Permanently reactivating lustre-OST0001 [ 6721.512484] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6721.519763] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6721.529940] LustreError: Skipped 3 previous similar messages [ 6721.546983] Lustre: Skipped 17 previous similar messages [ 6727.412775] Lustre: server umount lustre-OST0001 complete [ 6727.414593] Lustre: Skipped 7 previous similar messages [ 6731.767319] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6731.790449] LustreError: Skipped 40 previous similar messages [ 6742.000677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6742.020102] Lustre: Skipped 11 previous similar messages [ 6751.143894] LustreError: 238015:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206213 with bad export cookie 14787142397137508110 [ 6751.159792] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6751.168069] LustreError: 238015:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6756.543584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6760.011733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6763.728360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6767.186229] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6774.360309] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 18:37:15 (1776206235) [ 6775.451505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6782.228137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6782.762678] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6782.828212] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6782.834509] Lustre: Skipped 5 previous similar messages [ 6790.133313] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 6799.955725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6800.066393] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6805.115850] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 6811.539196] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6816.726052] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6823.399789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6823.491975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6823.660448] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6823.698465] Lustre: Permanently deactivating lustre-OST0000 [ 6828.344629] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 6831.136419] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6837.358634] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6844.391837] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6844.431952] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6850.344980] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 6853.376736] Lustre: lustre-OST0001: deleting orphan objects from 0x0:67 to 0x0:97 [ 6857.458929] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6859.091285] Lustre: Permanently reactivating lustre-OST0000 [ 6859.105676] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 6859.112344] Lustre: Skipped 4 previous similar messages [ 6859.948053] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6859.951500] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6859.974838] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 6860.004134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6860.044761] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 6905.311459] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6909.179036] LustreError: 244854:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206371 with bad export cookie 14787142397137509720 [ 6909.180101] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6909.197615] LustreError: 244854:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6920.636908] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 6923.543241] Key type lgssc unregistered [ 6923.789607] LNet: 249829:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6923.799864] LNet: Removed LNI 192.168.202.119@tcp [ 6924.647855] Key type .llcrypt unregistered [ 6924.652404] Key type ._llcrypt unregistered [ 6946.778355] alg: No test for adler32 (adler32-zlib) [ 6947.534797] Key type ._llcrypt registered [ 6947.539303] Key type .llcrypt registered [ 6947.689244] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 6948.670803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6949.405354] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 6949.691616] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 6949.699584] LNet: Accept secure, port 988 [ 6951.359129] Key type lgssc registered [ 6952.415513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6955.247778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6962.985628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6964.403743] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6964.460822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6969.518396] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 6969.826652] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6974.946594] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6977.200644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6977.278955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6977.364396] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6977.421057] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6977.432534] Lustre: Permanently deactivating lustre-MDT0001 [ 6977.674240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6979.214873] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6979.217806] LustreError: 251394:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6979.239674] LustreError: 251394:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6979.254940] LustreError: 251394:0:(lod_dev.c:437:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -108 [ 6981.708861] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 6989.835605] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6995.707749] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7002.153394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7002.499137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7006.808318] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 7007.601534] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7007.629944] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:163 to 0x280000400:193 [ 7012.836210] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7014.908532] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 7015.087923] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7021.717100] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7021.979447] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7027.156654] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 7027.203034] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:99 to 0x2c0000400:129 [ 7027.214102] Lustre: lustre-OST0001: deleting orphan objects from 0x0:67 to 0x0:129 [ 7034.837739] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7037.841584] Lustre: Permanently reactivating lustre-MDT0001 [ 7037.857098] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 7037.859253] Lustre: Skipped 3 previous similar messages [ 7042.740749] Lustre: lustre-MDT0001: Client e7a30ec6-0f3c-4f00-962a-e0ec7b3a53d6 (at 192.168.202.19@tcp) reconnecting [ 7048.113479] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7049.449553] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7055.143545] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 7055.316888] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7059.473687] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 7059.664344] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7062.047680] Lustre: Permanently deactivating lustre-MDT0001 [ 7062.065607] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 7062.076134] Lustre: Skipped 2 previous similar messages [ 7071.898431] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7085.027564] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7085.034049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7085.043422] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7085.056235] LustreError: Skipped 4 previous similar messages [ 7085.182189] Lustre: server umount lustre-MDT0000 complete [ 7086.564097] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7086.582909] Lustre: Skipped 2 previous similar messages [ 7089.287794] LustreError: 251345:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206551 with bad export cookie 16752099811003457092 [ 7089.294768] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7089.304033] LustreError: 251345:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7089.576748] Lustre: server umount lustre-MDT0001 complete [ 7094.859204] Lustre: server umount lustre-OST0000 complete [ 7104.597678] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 18:42:46 (1776206566) [ 7114.774984] Lustre: server umount lustre-OST0001 complete [ 7120.700111] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 7128.308942] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7137.511680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7143.384254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7148.508726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7155.782843] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7165.015202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7165.127886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7165.305470] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7165.311692] Lustre: Skipped 2 previous similar messages [ 7165.334637] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7165.387694] Lustre: lustre-MDT0000: new disk, initializing [ 7165.455563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7165.477786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7170.148729] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 7180.844881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7180.928432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7181.019484] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7181.051988] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7181.055664] Lustre: Skipped 1 previous similar message [ 7181.114414] Lustre: lustre-MDT0001: new disk, initializing [ 7181.194467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7181.226284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7181.244175] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7186.923485] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 7199.342300] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7207.261500] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7215.162585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7215.263920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7215.272394] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7215.595222] Lustre: lustre-OST0000: new disk, initializing [ 7215.609728] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7215.712122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7220.917372] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 7223.371839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7223.391860] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7231.421379] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7233.582627] LustreError: 261954:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7243.165967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7243.293410] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7243.302393] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 7243.479743] Lustre: lustre-OST0001: new disk, initializing [ 7243.483444] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 7243.622303] LustreError: 261954:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 7243.670209] LustreError: 261954:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7248.985236] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 7251.573738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7251.598647] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7253.759192] LustreError: 261954:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 7259.351807] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7261.065386] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7264.948353] Lustre: lustre-OST0001: Not available for connect from 192.168.202.19@tcp (stopping) [ 7267.303096] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7267.313381] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7267.317269] Lustre: Skipped 1 previous similar message [ 7267.331874] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7268.739575] Lustre: server umount lustre-OST0001 complete [ 7270.024764] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7270.039411] LustreError: Skipped 4 previous similar messages [ 7287.777766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7287.785848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7287.790276] Lustre: Skipped 1 previous similar message [ 7287.807749] Lustre: Skipped 2 previous similar messages [ 7291.812538] Lustre: server umount lustre-OST0000 complete [ 7298.016591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7298.019140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7298.040980] Lustre: Skipped 1 previous similar message [ 7303.139140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7303.141359] Lustre: Skipped 2 previous similar messages [ 7309.284679] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7309.468111] Lustre: server umount lustre-MDT0000 complete [ 7313.350439] LustreError: 259856:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206775 with bad export cookie 16752099811003459878 [ 7313.352933] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7313.371606] LustreError: 259856:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7332.983404] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 7335.474364] Key type lgssc unregistered [ 7335.745166] LNet: 265070:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7335.761817] LNet: Removed LNI 192.168.202.119@tcp [ 7336.472770] Key type .llcrypt unregistered [ 7336.474685] Key type ._llcrypt unregistered [ 7371.912371] alg: No test for adler32 (adler32-zlib) [ 7372.671288] Key type ._llcrypt registered [ 7372.676103] Key type .llcrypt registered [ 7372.809783] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 7385.960853] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7387.206439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7387.636225] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 7387.841581] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 7387.846453] LNet: Accept secure, port 988 [ 7389.511182] Key type lgssc registered [ 7390.678322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7399.825962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7407.606349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7414.700700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7421.090526] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7427.901624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7427.950877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7429.247832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7429.294991] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7429.394409] Lustre: lustre-MDT0000: new disk, initializing [ 7429.503761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7429.541682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7434.837343] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 7447.511488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7447.638919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7447.816973] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7447.872049] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7447.875986] Lustre: Skipped 1 previous similar message [ 7447.980269] Lustre: lustre-MDT0001: new disk, initializing [ 7448.119179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7448.149762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7448.168498] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7455.342968] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 7466.504317] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7472.343561] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7479.817978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7479.892497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7479.898229] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7480.080720] Lustre: lustre-OST0000: new disk, initializing [ 7480.082835] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7480.130515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7485.055478] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 7489.567620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7489.579534] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7497.071095] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7503.295768] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7503.489170] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7508.871460] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7509.035486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7515.105877] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7515.109534] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7515.117971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7515.120516] Lustre: Skipped 1 previous similar message [ 7520.232270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7520.253442] Lustre: Skipped 1 previous similar message [ 7525.346765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7525.352228] Lustre: Skipped 1 previous similar message [ 7525.855156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7525.977691] Lustre: server umount lustre-OST0000 complete [ 7529.952263] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7529.957336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7529.976992] Lustre: Skipped 1 previous similar message [ 7529.984261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7535.584623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7535.587253] Lustre: Skipped 2 previous similar messages [ 7536.071558] Lustre: server umount lustre-MDT0000 complete [ 7540.716825] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7540.726076] LustreError: Skipped 1 previous similar message [ 7541.622049] LustreError: 269165:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776207003 with bad export cookie 16496736443735480329 [ 7541.631316] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7541.633019] LustreError: 269165:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7542.312599] Lustre: server umount lustre-MDT0001 complete [ 7552.419621] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 18:50:13 (1776207013) [ 7559.724823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7560.366320] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7560.525187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7565.341556] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 7565.792380] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7570.912427] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7573.891612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7574.445250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7578.882806] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 7587.424867] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7593.383901] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7600.001425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7600.391221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7605.497519] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 7614.276832] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7640.551662] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7640.572346] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7640.599033] Lustre: Skipped 1 previous similar message [ 7640.610167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7642.826940] Lustre: server umount lustre-OST0000 complete [ 7646.966688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7659.944289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7660.384357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7662.453893] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7662.478273] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 7662.490484] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 7666.266901] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 7673.788787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7673.814281] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.119@tcp (at 0@lo) [ 7673.818520] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 7674.709211] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7688.161300] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7688.169096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7688.183401] Lustre: Skipped 1 previous similar message [ 7688.187721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7688.191438] Lustre: Skipped 2 previous similar messages [ 7690.326126] Lustre: server umount lustre-OST0000 complete [ 7697.376217] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7697.394045] LustreError: Skipped 1 previous similar message [ 7697.400157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7697.436220] Lustre: Skipped 1 previous similar message [ 7702.416182] Lustre: server umount lustre-MDT0000 complete [ 7704.544582] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7704.562024] LustreError: Skipped 1 previous similar message [ 7706.942269] LustreError: 278075:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776207169 with bad export cookie 16496736443735481183 [ 7706.946452] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7706.953955] LustreError: 278075:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7720.134943] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 7722.863379] Key type lgssc unregistered [ 7723.157938] LNet: 279167:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7723.166979] LNet: Removed LNI 192.168.202.119@tcp [ 7723.870640] Key type .llcrypt unregistered [ 7723.873910] Key type ._llcrypt unregistered [ 7746.742421] alg: No test for adler32 (adler32-zlib) [ 7747.498065] Key type ._llcrypt registered [ 7747.500226] Key type .llcrypt registered [ 7747.633277] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7749.032661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7749.560730] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 7749.819570] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 7749.823653] LNet: Accept secure, port 988 [ 7751.543355] Key type lgssc registered [ 7752.705317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7760.848779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7762.319104] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7762.465242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7767.471874] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 7767.524892] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7772.645809] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7776.660150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7777.040413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7781.544139] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 7788.136935] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7793.447451] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7799.825960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7800.173545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7805.197245] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 7813.503225] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:97 [ 7814.207228] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7838.176478] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7838.187877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7838.243162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7839.200917] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7839.207694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7839.210806] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7839.235670] Lustre: Skipped 1 previous similar message [ 7842.280669] Lustre: server umount lustre-OST0000 complete [ 7848.930082] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7848.940156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7848.958374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7853.246425] Lustre: server umount lustre-MDT0000 complete [ 7854.572258] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7854.595490] LustreError: Skipped 1 previous similar message [ 7857.928192] LustreError: 280614:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776207320 with bad export cookie 18368317122228080746 [ 7857.939810] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7858.306416] Lustre: server umount lustre-MDT0001 complete [ 7869.850814] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 7872.745702] Key type lgssc unregistered [ 7873.052956] LNet: 285058:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7873.063326] LNet: Removed LNI 192.168.202.119@tcp [ 7873.949274] Key type .llcrypt unregistered [ 7873.953952] Key type ._llcrypt unregistered [ 7890.339809] alg: No test for adler32 (adler32-zlib) [ 7891.094418] Key type ._llcrypt registered [ 7891.096906] Key type .llcrypt registered [ 7891.206419] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7892.189178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7892.888157] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 7893.179663] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 7893.185152] LNet: Accept secure, port 988 [ 7894.895141] Key type lgssc registered [ 7895.984368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7903.937778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7905.470367] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7905.613741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7909.926793] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 7910.883498] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7916.015705] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7917.562133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7917.840304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7922.212618] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 7930.462829] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7938.423790] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7948.940538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7949.156750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7953.989112] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 7955.312075] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 7961.375048] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7975.916071] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7975.920302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7975.933037] LustreError: Skipped 1 previous similar message [ 7975.937186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7975.969821] Lustre: Skipped 1 previous similar message [ 7977.728275] Lustre: server umount lustre-OST0000 complete [ 7984.608831] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7984.619657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7984.638187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7984.640946] Lustre: Skipped 1 previous similar message [ 7986.150099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7986.152143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7991.266597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7991.273860] Lustre: Skipped 2 previous similar messages [ 7996.383297] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7996.507300] Lustre: server umount lustre-MDT0000 complete [ 7997.416731] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7997.427409] LustreError: Skipped 1 previous similar message [ 8000.500066] LustreError: 286319:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776207462 with bad export cookie 9471956158350971621 [ 8000.531533] LustreError: 286319:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8000.537599] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8000.913535] Lustre: server umount lustre-MDT0001 complete [ 8013.058600] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 8015.612930] Key type lgssc unregistered [ 8015.901716] LNet: 290168:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8015.911325] LNet: Removed LNI 192.168.202.119@tcp [ 8016.959340] Key type .llcrypt unregistered [ 8016.964285] Key type ._llcrypt unregistered [ 8031.821991] alg: No test for adler32 (adler32-zlib) [ 8032.577310] Key type ._llcrypt registered [ 8032.582296] Key type .llcrypt registered [ 8032.751474] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8034.132435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8034.674648] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 8035.006669] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 8035.026297] LNet: Accept secure, port 988 [ 8036.807219] Key type lgssc registered [ 8037.966801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8045.873952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8047.292613] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8047.382380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8051.672954] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 8052.704650] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8057.828090] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8061.205429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8061.532702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8066.315625] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 8075.452245] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8082.302589] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8088.859850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8089.158511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8093.863829] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 8098.366158] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 8100.917712] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8112.615275] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8112.622942] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8112.632475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8113.651535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8113.662823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8113.691990] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8115.404595] Lustre: server umount lustre-OST0000 complete [ 8122.848751] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8122.858910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8122.874129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8122.884248] Lustre: Skipped 1 previous similar message [ 8126.830481] Lustre: server umount lustre-MDT0000 complete [ 8129.000395] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8129.020420] LustreError: Skipped 1 previous similar message [ 8132.243556] LustreError: 291417:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776207594 with bad export cookie 17827834270795009272 [ 8132.260408] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8132.266155] LustreError: 291417:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8132.702929] Lustre: server umount lustre-MDT0001 complete [ 8145.214987] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 8148.400914] Key type lgssc unregistered [ 8148.695744] LNet: 295091:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8148.706596] LNet: Removed LNI 192.168.202.119@tcp [ 8149.406939] Key type .llcrypt unregistered [ 8149.410591] Key type ._llcrypt unregistered [ 8174.883353] alg: No test for adler32 (adler32-zlib) [ 8175.636619] Key type ._llcrypt registered [ 8175.642336] Key type .llcrypt registered [ 8175.766813] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8177.315740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8177.703820] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 8177.935307] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 8177.941812] LNet: Accept secure, port 988 [ 8179.639400] Key type lgssc registered [ 8180.865453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8189.187845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8190.641151] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8190.694190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8194.807877] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 8196.065983] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8201.191355] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8204.004423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8204.409682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8209.178662] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 8217.241728] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8223.637137] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8230.989521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8231.234359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8236.550215] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 8241.473638] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 8244.144817] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8270.306476] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8270.321789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8270.339253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8272.358036] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8272.360949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8272.363399] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8272.385290] Lustre: Skipped 1 previous similar message [ 8274.274234] Lustre: server umount lustre-OST0000 complete [ 8281.059405] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8281.065050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8281.079952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8284.370171] Lustre: server umount lustre-MDT0000 complete [ 8287.712874] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8287.741653] LustreError: Skipped 1 previous similar message [ 8288.737746] LustreError: 296535:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776207750 with bad export cookie 5844782847316938656 [ 8288.740845] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8288.746741] LustreError: 296535:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8289.126875] Lustre: server umount lustre-MDT0001 complete [ 8302.328977] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 8305.653205] Key type lgssc unregistered [ 8306.112203] LNet: 301045:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8306.141570] LNet: Removed LNI 192.168.202.119@tcp [ 8307.349732] Key type .llcrypt unregistered [ 8307.352629] Key type ._llcrypt unregistered [ 8325.385787] alg: No test for adler32 (adler32-zlib) [ 8326.145492] Key type ._llcrypt registered [ 8326.148103] Key type .llcrypt registered [ 8326.255974] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8327.371610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8328.157557] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 8328.458129] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 8328.467879] LNet: Accept secure, port 988 [ 8330.175481] Key type lgssc registered [ 8331.374435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8338.683752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8340.091437] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8340.188386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8344.174470] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 8345.568455] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8350.689142] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8352.523810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8352.827210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8357.086888] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 8364.174563] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8369.062962] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8375.537512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8375.805322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8381.139855] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 8387.956317] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8390.189371] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 8403.427662] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8403.436050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8403.447495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8405.486868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8405.488484] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8405.508787] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8409.445948] Lustre: server umount lustre-OST0000 complete [ 8414.177611] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8414.186156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8414.197238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8414.204964] Lustre: Skipped 1 previous similar message [ 8419.349193] Lustre: server umount lustre-MDT0000 complete [ 8420.832363] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8420.845647] LustreError: Skipped 1 previous similar message [ 8423.330555] LustreError: 302329:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776207885 with bad export cookie 16725157650174113823 [ 8423.337949] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8423.341572] LustreError: 302329:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8423.608957] Lustre: server umount lustre-MDT0001 complete [ 8434.687462] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 8436.838770] Key type lgssc unregistered [ 8437.046647] LNet: 306188:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8437.050665] LNet: Removed LNI 192.168.202.119@tcp [ 8437.771816] Key type .llcrypt unregistered [ 8437.774285] Key type ._llcrypt unregistered [ 8451.916960] alg: No test for adler32 (adler32-zlib) [ 8452.674342] Key type ._llcrypt registered [ 8452.682205] Key type .llcrypt registered [ 8452.780130] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8453.922641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8454.477840] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 8454.693508] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 8454.697236] LNet: Accept secure, port 988 [ 8456.359148] Key type lgssc registered [ 8457.570935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8464.886288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8466.229814] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8466.278879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8471.049896] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 8471.531700] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8476.655245] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8481.093664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8481.459787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8487.318930] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 8496.654307] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8502.113606] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8508.476943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8508.850837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8514.293559] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 8517.059708] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:257 [ 8522.167331] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8532.448118] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8532.453189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8532.457651] LustreError: Skipped 1 previous similar message [ 8532.458472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8532.478211] Lustre: Skipped 1 previous similar message [ 8537.569085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8537.579973] Lustre: Skipped 2 previous similar messages [ 8542.690794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8544.223520] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8544.431296] Lustre: server umount lustre-OST0000 complete [ 8548.326513] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8548.331551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8548.359350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8548.362561] Lustre: Skipped 1 previous similar message [ 8552.928596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8552.930305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8552.948680] Lustre: Skipped 1 previous similar message [ 8554.156728] Lustre: server umount lustre-MDT0000 complete [ 8557.667882] LustreError: 307438:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776208019 with bad export cookie 13196376694013737882 [ 8557.683322] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8557.924054] Lustre: server umount lustre-MDT0001 complete [ 8569.760940] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 8572.952755] Key type lgssc unregistered [ 8573.422793] LNet: 311113:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8573.434303] LNet: Removed LNI 192.168.202.119@tcp [ 8574.286086] Key type .llcrypt unregistered [ 8574.288085] Key type ._llcrypt unregistered [ 8589.934265] alg: No test for adler32 (adler32-zlib) [ 8590.689407] Key type ._llcrypt registered [ 8590.690980] Key type .llcrypt registered [ 8590.794507] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8608.051600] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 8619.920470] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8620.806521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8621.582470] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 8621.802664] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 8621.805751] LNet: Accept secure, port 988 [ 8623.479724] Key type lgssc registered [ 8624.572056] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8633.074859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8639.578703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8645.606518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8652.150350] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8659.609782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8659.678380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8661.128949] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8661.156537] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8661.246515] Lustre: lustre-MDT0000: new disk, initializing [ 8661.312301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8661.332871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8666.363849] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 8676.032630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8676.090290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8676.170805] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8676.225482] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8676.231247] Lustre: Skipped 1 previous similar message [ 8676.308951] Lustre: lustre-MDT0001: new disk, initializing [ 8676.381691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8676.407541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8676.414451] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8680.378655] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 8689.325365] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8695.522517] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8701.962844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8702.027666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8702.032443] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8702.304647] Lustre: lustre-OST0000: new disk, initializing [ 8702.309153] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8702.441238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8707.242529] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 8711.756150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8711.768936] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8717.414556] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8723.266894] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8723.424824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8728.839917] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 8729.028240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8732.640483] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8732.640696] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8732.672190] LustreError: Skipped 1 previous similar message [ 8732.701057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8737.917338] Lustre: server umount lustre-OST0000 complete [ 8742.884495] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8742.896096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8742.911364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8742.918183] Lustre: Skipped 2 previous similar messages [ 8742.926639] Lustre: Skipped 1 previous similar message [ 8748.000846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8748.006325] Lustre: Skipped 2 previous similar messages [ 8748.212327] Lustre: server umount lustre-MDT0000 complete [ 8749.034412] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8749.053266] LustreError: Skipped 1 previous similar message [ 8753.179671] LustreError: 314859:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776208215 with bad export cookie 3266187244253984107 [ 8753.180404] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8753.197369] LustreError: 314859:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8753.570730] Lustre: server umount lustre-MDT0001 complete [ 8763.271464] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 19:10:24 (1776208224) [ 8770.631697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8771.089280] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8771.160176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8776.163208] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8776.712431] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 8781.281886] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8786.402825] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8786.509866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8786.893863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8791.743303] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 8799.598668] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8805.277763] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8811.690915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8812.078538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8816.175545] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 8823.428207] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8831.458412] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8831.462965] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8831.466267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8831.472532] Lustre: Skipped 1 previous similar message [ 8836.584865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8836.599513] Lustre: Skipped 1 previous similar message [ 8837.132929] Lustre: server umount lustre-OST0000 complete [ 8841.707184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8846.816896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8846.827606] Lustre: Skipped 4 previous similar messages [ 8847.115718] Lustre: server umount lustre-MDT0000 complete [ 8850.608888] LustreError: 319397:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776208312 with bad export cookie 3266187244253984961 [ 8850.611837] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8850.620524] LustreError: 319397:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8862.210288] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 8864.707394] Key type lgssc unregistered [ 8864.971666] LNet: 323073:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8864.985312] LNet: Removed LNI 192.168.202.119@tcp [ 8865.832382] Key type .llcrypt unregistered [ 8865.837772] Key type ._llcrypt unregistered [ 8890.411634] alg: No test for adler32 (adler32-zlib) [ 8891.165137] Key type ._llcrypt registered [ 8891.173939] Key type .llcrypt registered [ 8891.267561] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 8904.147910] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8905.152084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8906.019310] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 8906.181471] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 8906.184720] LNet: Accept secure, port 988 [ 8907.831253] Key type lgssc registered [ 8908.842831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8916.583410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8922.302394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8927.682156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8933.255728] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8937.216200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8941.647473] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8947.491992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8947.568850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8948.754409] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8948.769748] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8948.817652] Lustre: lustre-MDT0000: new disk, initializing [ 8948.869913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8948.888897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8953.303440] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 8963.632866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8963.698401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8963.771314] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8963.809214] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8963.812891] Lustre: Skipped 1 previous similar message [ 8963.881679] Lustre: lustre-MDT0001: new disk, initializing [ 8963.956347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8963.995793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8964.013137] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8968.092668] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 8976.397690] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8981.762655] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8988.265809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8988.356304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8988.360946] Lustre: lustre-OST2710-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8988.570251] Lustre: lustre-OST2710: new disk, initializing [ 8988.573557] Lustre: srv-lustre-OST2710: No data found on store. Initialize space: rc = -61 [ 8988.655968] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8993.350121] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 8994.897601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8994.908942] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9002.400292] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 9008.818296] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9008.905602] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9008.911686] Lustre: lustre-OST03e8-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 9009.004708] Lustre: lustre-OST03e8: new disk, initializing [ 9009.008576] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space: rc = -61 [ 9009.089306] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9012.845389] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 9015.866322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9015.873921] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9019.602248] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 9025.777677] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 40 [ 9025.992615] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9030.731378] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 40 [ 9030.947393] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9035.910294] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 40 [ 9036.109308] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9039.909888] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 40 [ 9040.159394] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9045.987140] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9045.993312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9046.001254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9046.665531] Lustre: server umount lustre-MDT0000 complete [ 9047.016486] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9047.017456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9047.043925] Lustre: Skipped 1 previous similar message [ 9050.428970] LustreError: 327017:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776208512 with bad export cookie 724722227542004432 [ 9050.450902] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9050.918544] Lustre: server umount lustre-MDT0001 complete [ 9059.231191] Lustre: 324817:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776208514/real 1776208514] req@0000000056e4a8a2 x1862489465713728/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776208521 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9059.256746] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9059.277148] Lustre: Skipped 1 previous similar message [ 9060.831135] Lustre: 332508:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776208516/real 1776208516] req@000000008b62b608 x1862489465713856/t0(0) o39->lustre-MDT0001-lwp-OST2710@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776208522 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9060.877631] Lustre: server umount lustre-OST2710 complete [ 9062.372436] Lustre: 324815:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776208517/real 1776208517] req@000000002cd152b8 x1862489465713920/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776208524 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9064.415262] Lustre: 324818:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776208519/real 1776208519] req@00000000f8781799 x1862489465714112/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776208526 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9064.439835] Lustre: 324818:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 9065.704128] Lustre: server umount lustre-OST03e8 complete [ 9085.486701] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 9096.275637] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 9105.812612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9111.539492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9117.052124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9122.806872] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9129.594456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9129.675584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9129.927792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9129.956447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9130.052879] Lustre: lustre-MDT0000: new disk, initializing [ 9130.154377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9130.176610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9134.718496] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 9145.758456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9145.854732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9145.995856] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9146.128779] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9146.144148] Lustre: Skipped 1 previous similar message [ 9146.317549] Lustre: lustre-MDT0001: new disk, initializing [ 9146.437155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9146.540798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9146.553299] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9151.377348] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 9161.888511] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9167.566067] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9173.756929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9173.826463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9173.835624] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 9174.095044] Lustre: lustre-OST0000: new disk, initializing [ 9174.102401] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9174.199611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9179.402311] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 9183.329946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9183.344177] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9190.223969] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9195.073970] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 9195.316942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9199.631969] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 9199.829215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9204.192258] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9204.207917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9204.216695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9204.226674] Lustre: Skipped 1 previous similar message [ 9204.245069] Lustre: Skipped 1 previous similar message [ 9208.003676] Lustre: server umount lustre-OST0000 complete [ 9212.896867] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9212.910695] LustreError: Skipped 1 previous similar message [ 9212.916273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9212.957338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9217.539421] Lustre: server umount lustre-MDT0000 complete [ 9219.556945] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9219.572993] LustreError: Skipped 4 previous similar messages [ 9220.579849] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9220.594090] LustreError: Skipped 1 previous similar message [ 9222.209450] LustreError: 335617:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776208684 with bad export cookie 724722227542006042 [ 9222.210183] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9222.233685] LustreError: 335617:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9231.998633] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 19:18:13 (1776208693) [ 9234.205646] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9236.506637] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 19:18:17 (1776208697) [ 9240.015592] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 9247.357943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9251.902494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9255.795819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9260.556721] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9261.864985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9269.554981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9269.786910] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9269.830406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9270.018444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9276.029659] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 9284.417332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9284.497517] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9284.533339] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9289.170783] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 9296.606206] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9302.143292] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9308.296725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9308.352442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9308.559380] Lustre: Denying initial registration attempt from nid 192.168.202.119@tcp, specified as failover [ 9308.570460] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9308.581062] LustreError: 343643:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -99 [ 9308.600357] LustreError: 343643:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 9308.615931] LustreError: 343643:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 9308.727633] Lustre: server umount lustre-OST0000 complete [ 9308.732193] Lustre: Skipped 1 previous similar message [ 9308.736105] LustreError: 343643:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9315.300753] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9315.313143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9315.322735] Lustre: Skipped 1 previous similar message [ 9315.327094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9315.330144] Lustre: Skipped 2 previous similar messages [ 9322.466886] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9322.485044] LustreError: Skipped 1 previous similar message [ 9322.897319] LustreError: 341684:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776208784 with bad export cookie 724722227542006896 [ 9322.900212] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9322.905814] LustreError: 341684:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9335.825490] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 9338.534202] Key type lgssc unregistered [ 9338.849283] LNet: 344718:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9338.864641] LNet: Removed LNI 192.168.202.119@tcp [ 9339.603751] Key type .llcrypt unregistered [ 9339.606573] Key type ._llcrypt unregistered [ 9353.620174] alg: No test for adler32 (adler32-zlib) [ 9354.378638] Key type ._llcrypt registered [ 9354.381429] Key type .llcrypt registered [ 9354.537099] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 9355.578188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9356.171327] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 9356.379266] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 9356.383855] LNet: Accept secure, port 988 [ 9358.071178] Key type lgssc registered [ 9359.182398] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9366.946582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9371.236446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9374.652582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9378.742484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9379.968515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9393.002611] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 9402.022878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9403.314279] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9403.351378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9403.523158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9408.495300] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 9417.470766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9417.535889] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9417.556392] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9417.726080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9422.690789] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 9432.466216] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9439.867994] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9448.108686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9448.247767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9448.501791] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9448.861394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9454.070073] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 9462.172710] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9466.848452] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9466.860415] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9466.878873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9467.362193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9470.485702] Lustre: server umount lustre-OST0000 complete [ 9477.605048] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9477.610054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9477.624962] Lustre: Skipped 1 previous similar message [ 9477.628940] Lustre: Skipped 2 previous similar messages [ 9480.856358] Lustre: server umount lustre-MDT0000 complete [ 9482.724345] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9482.752556] LustreError: Skipped 1 previous similar message [ 9485.318922] LustreError: 347589:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776208947 with bad export cookie 12731034570184991875 [ 9485.320918] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9485.347318] LustreError: 347589:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9485.589669] Lustre: server umount lustre-MDT0001 complete [ 9496.782518] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 9499.489910] Key type lgssc unregistered [ 9499.764731] LNet: 351293:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9499.770590] LNet: Removed LNI 192.168.202.119@tcp [ 9500.521175] Key type .llcrypt unregistered [ 9500.529263] Key type ._llcrypt unregistered [ 9526.214695] alg: No test for adler32 (adler32-zlib) [ 9526.977362] Key type ._llcrypt registered [ 9526.984126] Key type .llcrypt registered [ 9527.134768] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 9528.426402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9529.048398] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 9529.244705] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 9529.251600] LNet: Accept secure, port 988 [ 9530.919665] Key type lgssc registered [ 9532.222318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9542.251613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9544.043579] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9544.097282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9549.291421] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9549.528487] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 9554.413609] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9559.527865] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9562.013433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9562.342078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9568.432706] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 9578.996327] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9585.929159] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9592.182677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9592.502434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9597.515187] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 9605.370503] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9618.032623] Lustre: server umount lustre-MDT0000 complete [ 9618.918774] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9618.935906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9618.954434] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9618.975711] LustreError: Skipped 1 previous similar message [ 9622.501565] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9622.507286] LustreError: 11-0: lustre-MDT0000-lwp-MDT0001: operation mds_disconnect to node 0@lo failed: rc = -107 [ 9622.514303] LustreError: 352761:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776209084 with bad export cookie 2690924393788740641 [ 9622.515412] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9622.717980] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.19@tcp (stopping) [ 9627.632911] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9627.632991] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9627.644360] LustreError: Skipped 3 previous similar messages [ 9627.689944] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9628.705643] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9628.713068] Lustre: Skipped 1 previous similar message [ 9628.864244] Lustre: server umount lustre-MDT0001 complete [ 9637.485944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9647.816526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9656.291135] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0x2558157aa1361421 to 0x2558157aa1362ffe [ 9656.330468] Lustre: MGC192.168.202.119@tcp: Connection restored to 192.168.202.119@tcp (at 0@lo) [ 9656.859886] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9656.905725] LustreError: Skipped 2 previous similar messages [ 9657.099540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9657.253313] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:193 [ 9664.625184] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 9675.290065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9675.680806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9679.788893] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 9681.275797] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 9690.882074] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9692.997480] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9699.495294] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9701.430636] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9708.517348] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9708.525349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9708.557142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9713.795607] Lustre: server umount lustre-OST0000 complete [ 9720.801674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9720.806468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9720.821805] Lustre: Skipped 2 previous similar messages [ 9720.834485] Lustre: Skipped 3 previous similar messages [ 9724.684218] Lustre: server umount lustre-MDT0000 complete [ 9725.922482] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9725.936351] LustreError: Skipped 9 previous similar messages [ 9728.494987] LustreError: 352761:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776209190 with bad export cookie 2690924393788747774 [ 9728.501151] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9728.507050] LustreError: 352761:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9728.840785] Lustre: server umount lustre-MDT0001 complete [ 9742.189207] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 9745.086515] Key type lgssc unregistered [ 9745.455973] LNet: 359293:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9745.472065] LNet: Removed LNI 192.168.202.119@tcp [ 9746.148286] Key type .llcrypt unregistered [ 9746.150404] Key type ._llcrypt unregistered [ 9769.148929] alg: No test for adler32 (adler32-zlib) [ 9769.907547] Key type ._llcrypt registered [ 9769.917342] Key type .llcrypt registered [ 9770.095713] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 9771.639680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9772.281233] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 9772.529493] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 9772.532642] LNet: Accept secure, port 988 [ 9774.247126] Key type lgssc registered [ 9775.521683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9784.044453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9785.498742] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9785.581818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9789.784263] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 9790.946307] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9796.066566] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9798.693944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9799.160242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9804.769710] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 9812.794634] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9819.488692] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9824.737050] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9824.748883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9824.770657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9826.785986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9826.787293] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9829.303212] Lustre: server umount lustre-MDT0000 complete [ 9829.924702] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9829.938838] LustreError: Skipped 1 previous similar message [ 9832.894668] LustreError: 360736:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776209294 with bad export cookie 17101841093327151810 [ 9832.906134] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9833.256315] Lustre: server umount lustre-MDT0001 complete [ 9851.138058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9851.769807] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9851.844923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9852.159561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9856.639812] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 9866.136554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9866.300628] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9866.368879] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9866.773482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9872.984383] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 9882.540371] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9890.293246] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9899.870821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9900.174940] LustreError: 13b-9: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9900.193223] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9900.211850] LustreError: 365451:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 9900.222681] LustreError: 365451:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 9900.230759] LustreError: 365451:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 9900.370238] Lustre: server umount lustre-OST0000 complete [ 9900.372861] LustreError: 365451:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9907.943960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9908.154602] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9908.352424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9915.087208] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 9916.602906] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:225 [ 9922.851575] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9933.234402] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9933.370571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9933.380811] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 9933.482147] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9933.574393] Lustre: lustre-OST0001: new disk, initializing [ 9933.591127] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 9933.773307] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9939.898248] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [ 9942.654252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9942.662837] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9949.249204] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9953.249263] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9953.254064] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9953.255210] LustreError: Skipped 1 previous similar message [ 9953.255727] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9953.255731] Lustre: Skipped 1 previous similar message [ 9953.323622] Lustre: Skipped 1 previous similar message [ 9958.391599] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9958.404101] Lustre: Skipped 1 previous similar message [ 9963.493438] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9963.508285] Lustre: Skipped 1 previous similar message [ 9967.071532] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 9967.139043] Lustre: server umount lustre-OST0001 complete [ 9968.628166] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9968.661842] LustreError: Skipped 1 previous similar message [ 9972.704605] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9972.714792] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9972.736878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9977.413934] Lustre: server umount lustre-OST0000 complete [ 9983.969712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9984.003430] Lustre: Skipped 1 previous similar message [ 9987.486335] Lustre: server umount lustre-MDT0000 complete [ 9989.108653] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9989.134605] LustreError: Skipped 3 previous similar messages [ 9991.861960] LustreError: 363445:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776209453 with bad export cookie 17101841093327152356 [ 9991.881808] LustreError: 363445:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9991.902081] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10004.090095] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [10007.833304] Key type lgssc unregistered [10008.189955] LNet: 368822:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10008.201991] LNet: Removed LNI 192.168.202.119@tcp [10008.863502] Key type .llcrypt unregistered [10008.865434] Key type ._llcrypt unregistered [10044.989385] alg: No test for adler32 (adler32-zlib) [10045.744233] Key type ._llcrypt registered [10045.749034] Key type .llcrypt registered [10045.867229] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [10061.676830] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [10062.864134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10063.706702] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [10063.973131] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [10063.977762] LNet: Accept secure, port 988 [10065.623337] Key type lgssc registered [10066.785183] Lustre: Echo OBD driver; http://www.lustre.org/ [10075.006465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10082.081247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10087.903361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10093.378678] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10101.082281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10101.172858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10102.580113] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10102.628955] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [10102.774620] Lustre: lustre-MDT0000: new disk, initializing [10102.927675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10102.958938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10108.040590] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [10118.914348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10119.014142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10119.162234] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10119.210894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [10119.216262] Lustre: Skipped 1 previous similar message [10119.358790] Lustre: lustre-MDT0001: new disk, initializing [10119.510251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10119.562187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10119.578846] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10125.417916] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [10135.461911] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10141.247420] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10147.201750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10147.272150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10147.278741] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [10147.463820] Lustre: lustre-OST0000: new disk, initializing [10147.469812] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [10147.555166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10153.254253] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [10157.103629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10157.114629] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10162.892623] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10167.817170] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [10168.088801] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10172.970512] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [10173.219130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10177.513811] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10177.527131] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10177.533793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10177.540375] Lustre: Skipped 1 previous similar message [10182.639030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10182.663400] Lustre: Skipped 2 previous similar messages [10187.751756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10187.759618] Lustre: Skipped 1 previous similar message [10189.281189] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [10189.525582] Lustre: server umount lustre-OST0000 complete [10196.451303] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10196.464518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10196.476958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10197.986364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10199.953997] Lustre: server umount lustre-MDT0000 complete [10203.116465] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10203.134254] LustreError: Skipped 1 previous similar message [10203.937929] LustreError: 372936:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776209666 with bad export cookie 70418291975853229 [10203.941779] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10203.956308] LustreError: 372936:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [10204.356950] Lustre: server umount lustre-MDT0001 complete [10213.571374] Lustre: DEBUG MARKER: == conf-sanity test 60: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 19:34:34 (1776209674) [10219.867620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10226.285561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10250.898550] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [10259.632529] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [10269.048130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10275.014405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10280.939858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10287.551237] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10295.828509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10295.904437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10296.111571] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10296.137647] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [10296.217901] Lustre: lustre-MDT0000: new disk, initializing [10296.294717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10296.320597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10302.599468] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [10314.770444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10314.882098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10314.997343] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10315.049263] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [10315.057156] Lustre: Skipped 1 previous similar message [10315.133863] Lustre: lustre-MDT0001: new disk, initializing [10315.267169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10315.316731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10315.332448] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10320.350335] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [10329.358866] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10335.338856] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10340.968916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10341.049443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10341.056132] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [10341.293276] Lustre: lustre-OST0000: new disk, initializing [10341.300788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [10341.388580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10345.813912] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [10351.179056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10351.188677] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10355.362451] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10360.449933] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [10360.819248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10365.911936] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [10366.170706] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10372.066378] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10372.076341] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10372.085906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10372.102230] Lustre: Skipped 3 previous similar messages [10374.655189] Lustre: server umount lustre-OST0000 complete [10381.796826] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10381.809558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10381.829654] Lustre: Skipped 1 previous similar message [10381.842707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10384.855109] Lustre: server umount lustre-MDT0000 complete [10387.430348] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10387.448252] LustreError: Skipped 1 previous similar message [10388.449215] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10388.480680] LustreError: Skipped 1 previous similar message [10388.924966] LustreError: 380711:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776209851 with bad export cookie 70418291975854090 [10388.933416] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10388.937024] LustreError: 380711:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [10397.907730] Lustre: DEBUG MARKER: == conf-sanity test 61: large xattr ====================== 19:37:38 (1776209858) [10407.137857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10407.564190] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10407.618962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10412.403966] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [10413.027290] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10423.288310] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10423.322444] LustreError: Skipped 1 previous similar message [10424.424083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10429.836412] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [10437.507881] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10442.937438] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10448.926549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10449.361135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10449.367887] Lustre: Skipped 1 previous similar message [10453.548545] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [10460.881594] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10467.821798] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61.conf-sanity [10469.870476] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61.conf-sanity [10471.382967] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61.conf-sanity [10472.851514] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61.conf-sanity after remounting MDS [10474.706039] Lustre: Failing over lustre-MDT0000 [10474.916494] Lustre: server umount lustre-MDT0000 complete [10474.921172] Lustre: Skipped 1 previous similar message [10476.515545] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10476.528375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10476.541901] Lustre: Skipped 1 previous similar message [10476.554195] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10486.239647] Lustre: 371163:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776209941/real 1776209941] req@00000000e761a9ae x1862490679989952/t0(0) o400->MGC192.168.202.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776209948 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [10486.258920] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10495.145193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10503.668303] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xfa2d103ee91b60 to 0xfa2d103ee91f49 [10503.681258] Lustre: MGC192.168.202.119@tcp: Connection restored to 192.168.202.119@tcp (at 0@lo) [10504.045764] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10507.406905] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10509.125295] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [10509.308634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [10509.333594] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10509.377368] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [10518.932647] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10520.750649] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10524.455019] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61.conf-sanity [10526.553866] Lustre: Failing over lustre-MDT0000 [10526.779558] Lustre: server umount lustre-MDT0000 complete [10527.905144] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10527.930201] LustreError: Skipped 32 previous similar messages [10529.766285] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10529.766771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10529.811077] Lustre: Skipped 3 previous similar messages [10536.351264] Lustre: 371165:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776209991/real 1776209991] req@0000000064478ffd x1862490680000576/t0(0) o400->MGC192.168.202.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776209998 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [10536.367480] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10546.548939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10553.830328] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xfa2d103ee91f49 to 0xfa2d103ee92252 [10553.845296] Lustre: MGC192.168.202.119@tcp: Connection restored to (at 0@lo) [10553.857432] Lustre: Skipped 2 previous similar messages [10554.114310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10554.117374] Lustre: Skipped 1 previous similar message [10554.183732] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10556.553731] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10558.120389] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [10559.477955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [10559.509233] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10559.557404] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [10568.156528] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10569.813048] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10579.944553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10579.952318] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10579.961981] Lustre: Skipped 1 previous similar message [10579.977430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10579.989515] Lustre: Skipped 4 previous similar messages [10581.511477] Lustre: server umount lustre-MDT0000 complete [10588.281809] LustreError: 385350:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776210050 with bad export cookie 70418291975856722 [10588.287251] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10588.318021] LustreError: 385350:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [10600.927210] Lustre: 391159:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776210057/real 1776210057] req@00000000bbf8012f x1862490680013376/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776210063 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [10618.128076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10618.755747] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10618.772665] LustreError: Skipped 33 previous similar messages [10624.895512] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [10634.535537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10641.484858] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [10651.934579] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10658.568978] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10665.967313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10669.566717] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [10671.574583] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [10680.010501] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10696.174063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10696.180506] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10696.191883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10696.216773] Lustre: Skipped 1 previous similar message [10701.433110] Lustre: server umount lustre-OST0000 complete [10701.437356] Lustre: Skipped 2 previous similar messages [10715.874745] LustreError: 393937:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776210177 with bad export cookie 70418291975857450 [10715.883241] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10715.894035] LustreError: 393937:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [10728.442991] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [10730.850916] Key type lgssc unregistered [10731.082352] LNet: 395663:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10731.093468] LNet: Removed LNI 192.168.202.119@tcp [10731.836923] Key type .llcrypt unregistered [10731.838938] Key type ._llcrypt unregistered [10755.103244] alg: No test for adler32 (adler32-zlib) [10755.866620] Key type ._llcrypt registered [10755.871730] Key type .llcrypt registered [10756.002652] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [10757.110266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10757.903845] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [10758.089575] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [10758.095108] LNet: Accept secure, port 988 [10759.759479] Key type lgssc registered [10761.549132] Lustre: Echo OBD driver; http://www.lustre.org/ [10772.724560] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10772.745102] LustreError: 397151:0:(osd_handler.c:8215:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10772.762217] LustreError: 397151:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000-osd failed (-22) [10772.777048] LustreError: 397151:0:(obd_mount.c:200:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10772.813311] LustreError: 397151:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10772.837568] LustreError: 397151:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [10783.135937] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10783.142736] LustreError: 397548:0:(osd_handler.c:8215:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10783.155026] LustreError: 397548:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-22) [10783.164617] LustreError: 397548:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10783.169614] LustreError: 397548:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10783.176410] LustreError: 397548:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [10802.524360] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [10805.315266] Key type lgssc unregistered [10805.666562] LNet: 398508:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10805.671052] LNet: Removed LNI 192.168.202.119@tcp [10806.530943] Key type .llcrypt unregistered [10806.536453] Key type ._llcrypt unregistered [10827.829635] alg: No test for adler32 (adler32-zlib) [10828.593856] Key type ._llcrypt registered [10828.595654] Key type .llcrypt registered [10828.670987] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [10842.945794] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [10844.171703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10845.250767] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [10845.505638] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [10845.517188] LNet: Accept secure, port 988 [10847.199205] Key type lgssc registered [10848.513871] Lustre: Echo OBD driver; http://www.lustre.org/ [10857.297387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10863.496597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10869.318214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10876.064220] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10883.052513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10883.119222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10884.666655] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10884.747826] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [10884.904181] Lustre: lustre-MDT0000: new disk, initializing [10885.010531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10885.036872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10890.858690] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [10904.083541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10904.128994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10904.216444] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10904.286890] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [10904.301425] Lustre: Skipped 1 previous similar message [10904.506822] Lustre: lustre-MDT0001: new disk, initializing [10904.628373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10904.704893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10904.729271] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10910.056131] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [10920.723204] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10927.954719] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10937.355889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10937.492358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10937.504558] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [10937.817194] Lustre: lustre-OST0000: new disk, initializing [10937.820893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [10937.942879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10944.463022] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [10947.265101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10947.282833] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10958.230451] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10963.102678] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [10963.491426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10969.572857] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [10969.809815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10973.154601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10973.157444] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10973.191157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10973.206618] Lustre: Skipped 1 previous similar message [10978.282631] Lustre: server umount lustre-OST0000 complete [10983.392091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10983.401468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10983.414501] Lustre: Skipped 2 previous similar messages [10983.442170] Lustre: Skipped 1 previous similar message [10988.512342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10988.518649] Lustre: Skipped 1 previous similar message [10994.673912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10994.701401] Lustre: Skipped 3 previous similar messages [10997.215852] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [10997.482417] Lustre: server umount lustre-MDT0000 complete [10999.787279] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10999.797391] LustreError: Skipped 1 previous similar message [11001.856968] LustreError: 401800:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776210463 with bad export cookie 11111167885023773580 [11001.858112] LustreError: 401799:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776210463 with bad export cookie 11111167885023773580 [11001.860589] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11001.868635] LustreError: 401800:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [11002.240987] Lustre: server umount lustre-MDT0001 complete [11011.375669] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11013.105874] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 19:47:54 (1776210474) [11020.941708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11021.529798] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11021.628723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11026.913480] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11026.950588] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [11032.034512] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11035.454389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11035.799132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11040.532249] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [11048.215483] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11055.083697] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11062.012416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11062.377779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11066.861235] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [11076.046531] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11083.760135] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11083.853451] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11083.858753] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [11084.044483] Lustre: lustre-OST0001: new disk, initializing [11084.062608] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [11084.188574] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11089.445443] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [11090.534691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11090.540945] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11100.358904] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11106.275478] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11106.276606] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11106.277500] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11106.289966] Lustre: Skipped 1 previous similar message [11110.707478] Lustre: server umount lustre-OST0001 complete [11111.394970] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11111.421167] LustreError: Skipped 1 previous similar message [11121.639813] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11121.661582] LustreError: Skipped 4 previous similar messages [11129.318616] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11129.334283] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11129.361739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11129.373658] Lustre: Skipped 2 previous similar messages [11134.755268] Lustre: server umount lustre-OST0000 complete [11142.124490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11142.141407] Lustre: Skipped 2 previous similar messages [11145.794796] Lustre: server umount lustre-MDT0000 complete [11147.232670] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11147.244778] LustreError: Skipped 5 previous similar messages [11150.382526] LustreError: 406393:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776210612 with bad export cookie 11111167885023774434 [11150.389287] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11150.401026] LustreError: 406393:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [11171.557917] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [11174.405032] Key type lgssc unregistered [11174.768671] LNet: 411319:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11174.781361] LNet: Removed LNI 192.168.202.119@tcp [11175.545546] Key type .llcrypt unregistered [11175.549816] Key type ._llcrypt unregistered [11213.622353] alg: No test for adler32 (adler32-zlib) [11214.380382] Key type ._llcrypt registered [11214.383198] Key type .llcrypt registered [11214.477760] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [11229.925408] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [11231.629895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11232.298110] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [11232.619697] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [11232.623139] LNet: Accept secure, port 988 [11234.319163] Key type lgssc registered [11235.430538] Lustre: Echo OBD driver; http://www.lustre.org/ [11244.174087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11252.575946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11258.803345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11265.233384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11273.840113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11273.994166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11275.427831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [11275.473639] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [11275.606498] Lustre: lustre-MDT0000: new disk, initializing [11275.748280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11275.779150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11281.892175] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [11294.945620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11295.024631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11295.107990] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [11295.148146] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [11295.153264] Lustre: Skipped 1 previous similar message [11295.236441] Lustre: lustre-MDT0001: new disk, initializing [11295.363287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11295.403472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11295.420255] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11301.046650] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [11310.839776] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11320.915666] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11330.337354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11330.469729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11330.476151] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [11330.981404] Lustre: lustre-OST0000: new disk, initializing [11330.986935] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [11331.123542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11336.234309] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [11338.329585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11338.360536] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11347.563995] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11353.309896] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [11353.635278] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11358.726755] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [11358.988179] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11364.323826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11364.326108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11364.326235] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11364.349635] Lustre: Skipped 1 previous similar message [11364.389647] Lustre: Skipped 1 previous similar message [11369.441908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11369.453648] Lustre: Skipped 1 previous similar message [11374.562537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11374.572990] Lustre: Skipped 1 previous similar message [11375.071175] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [11375.247939] Lustre: server umount lustre-OST0000 complete [11379.680641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11379.699778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11379.711678] Lustre: Skipped 1 previous similar message [11384.802101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11384.818745] Lustre: Skipped 1 previous similar message [11385.566895] Lustre: server umount lustre-MDT0000 complete [11389.916616] LustreError: 416301:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776210852 with bad export cookie 6945671287739420211 [11389.922495] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11389.928311] LustreError: 416301:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [11390.459304] Lustre: server umount lustre-MDT0001 complete [11399.233636] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 19:54:20 (1776210860) [11409.765414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11421.125786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11421.817163] Lustre: lustre-MDT0000: new disk, initializing [11421.885239] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11422.036465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11427.300100] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11428.002897] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [11432.426789] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11437.539465] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11438.463963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11439.063977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11445.459038] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [11454.309352] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11461.258732] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11464.676389] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11464.676794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11464.694232] LustreError: Skipped 1 previous similar message [11464.696417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11464.707512] Lustre: Skipped 2 previous similar messages [11464.720807] Lustre: Skipped 1 previous similar message [11470.004457] Lustre: server umount lustre-MDT0000 complete [11470.828287] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11470.853938] LustreError: Skipped 1 previous similar message [11475.590966] LustreError: 420573:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776210937 with bad export cookie 6945671287739421065 [11475.612831] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11476.330922] Lustre: server umount lustre-MDT0001 complete [11487.332369] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 19:55:48 (1776210948) [11494.834795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11495.575808] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11495.698946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11500.774834] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [11510.769544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11516.926480] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [11525.569385] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11532.649677] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11540.646954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11541.042446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11541.045851] Lustre: Skipped 1 previous similar message [11545.339537] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [11553.117242] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11561.101983] Lustre: Permanently deactivating lustre-OST0000 [11561.112633] Lustre: Setting parameter lustre-OST0000-osc.osc.active in log lustre-client [11567.669931] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11569.151772] LustreError: 426189:0:(mgs_llog.c:1551:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002020 [11569.164153] LustreError: 426189:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [11569.178212] LustreError: 426189:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [11573.029909] LustreError: 426237:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [11573.039265] LustreError: 426237:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [11577.731488] Lustre: server umount lustre-OST0000 complete [11580.484261] LustreError: 426487:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [11580.502551] LustreError: 426487:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [11588.072875] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11588.083217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11588.113266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11588.125074] Lustre: Skipped 3 previous similar messages [11589.599691] Lustre: 426585:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776211045/real 1776211045] req@000000008d41b27d x1862491904721856/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776211051 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [11593.186747] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11593.206564] LustreError: Skipped 3 previous similar messages [11593.924305] LustreError: 426336:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776211056 with bad export cookie 6945671287739421618 [11593.927316] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11593.935289] LustreError: 426336:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 8 previous similar messages [11600.352140] Lustre: 426787:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776211056/real 1776211056] req@00000000e663a29f x1862491904723008/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776211062 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [11600.811596] Lustre: server umount lustre-MDT0001 complete [11600.822371] Lustre: Skipped 1 previous similar message [11611.670397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11616.294692] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [11619.143348] LustreError: 427699:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -6 [11625.777333] Lustre: 428046:0:(mgs_llog.c:1311:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.119@tcp [11634.658060] Lustre: 428145:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776211090/real 1776211090] req@0000000044e4ad2d x1862491904723520/t0(0) o251->MGC192.168.202.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776211096 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [11634.717030] Lustre: server umount MGS complete [11648.070329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11648.564803] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11648.585813] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11648.591535] Lustre: Skipped 1 previous similar message [11648.634866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11654.221914] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [11664.627116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11665.024035] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11670.437597] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [11680.035839] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11688.783145] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11692.320584] Lustre: Permanently reactivating lustre-OST0000 [11692.350978] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [11692.370612] Lustre: Skipped 2 previous similar messages [11720.672696] LustreError: 430628:0:(osp_dev.c:684:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11728.998068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11729.322573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11729.326275] Lustre: Skipped 1 previous similar message [11733.726282] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [11741.827328] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11752.944251] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [11752.960835] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11752.977745] Lustre: Skipped 1 previous similar message [11752.986070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11758.781585] Lustre: server umount lustre-OST0000 complete [11765.217745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11765.233229] Lustre: Skipped 2 previous similar messages [11770.336072] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11770.360338] LustreError: Skipped 4 previous similar messages [11774.457574] LustreError: 428755:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776211236 with bad export cookie 6945671287739422577 [11774.463439] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11774.467829] LustreError: 428755:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [11789.503548] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [11792.419646] Key type lgssc unregistered [11792.775623] LNet: 433216:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11792.791976] LNet: Removed LNI 192.168.202.119@tcp [11793.763069] Key type .llcrypt unregistered [11793.766913] Key type ._llcrypt unregistered [11815.085466] alg: No test for adler32 (adler32-zlib) [11815.839921] Key type ._llcrypt registered [11815.843229] Key type .llcrypt registered [11816.001653] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [11828.983576] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [11830.032568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11830.640351] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [11830.876379] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [11830.887851] LNet: Accept secure, port 988 [11832.559147] Key type lgssc registered [11833.616230] Lustre: Echo OBD driver; http://www.lustre.org/ [11842.578842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11849.198955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11856.155587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11864.013616] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11872.405383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11872.463213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11873.649351] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [11873.676313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [11873.758586] Lustre: lustre-MDT0000: new disk, initializing [11873.845144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11873.867121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11878.656178] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [11890.269242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11890.369920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11890.468548] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [11890.521049] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [11890.531733] Lustre: Skipped 1 previous similar message [11890.654336] Lustre: lustre-MDT0001: new disk, initializing [11890.778120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11890.831532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11890.849698] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11896.681318] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [11908.530795] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11916.495276] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11924.245174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11924.334977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11924.341937] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [11924.655756] Lustre: lustre-OST0000: new disk, initializing [11924.662345] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [11924.755277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11929.731041] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [11931.195190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11931.208226] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11939.713807] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11945.637725] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [11945.843835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11950.152322] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [11950.465408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11957.220851] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11957.236149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11957.240668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11957.284384] Lustre: Skipped 1 previous similar message [11958.454693] Lustre: server umount lustre-OST0000 complete [11967.458686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11967.469671] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11967.475154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11967.475164] Lustre: Skipped 1 previous similar message [11967.490578] Lustre: Skipped 1 previous similar message [11967.569308] LustreError: Skipped 1 previous similar message [11968.916274] Lustre: server umount lustre-MDT0000 complete [11972.577201] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11972.594855] LustreError: Skipped 1 previous similar message [11973.610067] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11973.625964] LustreError: Skipped 1 previous similar message [11974.149943] LustreError: 436502:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776211436 with bad export cookie 7498757812527421495 [11974.159272] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11974.175869] LustreError: 436502:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [11974.804602] Lustre: server umount lustre-MDT0001 complete [11985.210948] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11987.197771] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 20:04:08 (1776211448) [11993.269762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11993.698149] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11993.738632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11997.437149] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [11999.215583] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12004.324095] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12005.387503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12005.721707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12010.724618] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [12017.912978] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12024.628633] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12033.464478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12033.986534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12040.336808] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [12051.542320] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12053.811647] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12057.058157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12065.760142] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12065.773986] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12065.792979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12065.798601] Lustre: Skipped 1 previous similar message [12069.859673] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12069.869048] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12069.870783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12069.902648] Lustre: Skipped 1 previous similar message [12071.404387] Lustre: server umount lustre-OST0000 complete [12077.536208] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12077.549817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12077.568517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12081.762463] Lustre: server umount lustre-MDT0000 complete [12085.233086] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12085.269130] LustreError: Skipped 1 previous similar message [12087.125069] LustreError: 441902:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776211549 with bad export cookie 7498757812527422349 [12087.135807] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12087.144198] LustreError: 441902:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [12102.832741] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12106.116699] Key type lgssc unregistered [12106.405972] LNet: 444964:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12106.416354] LNet: Removed LNI 192.168.202.119@tcp [12107.079076] Key type .llcrypt unregistered [12107.081829] Key type ._llcrypt unregistered [12134.262363] alg: No test for adler32 (adler32-zlib) [12135.020581] Key type ._llcrypt registered [12135.030050] Key type .llcrypt registered [12135.167617] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12135.473349] Key type .llcrypt unregistered [12135.479329] Key type ._llcrypt unregistered [12153.375760] alg: No test for adler32 (adler32-zlib) [12154.158791] Key type ._llcrypt registered [12154.165248] Key type .llcrypt registered [12154.261942] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [12155.704992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12156.094166] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [12156.283763] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [12156.288693] LNet: Accept secure, port 988 [12157.951217] Key type lgssc registered [12159.083740] Lustre: Echo OBD driver; http://www.lustre.org/ [12167.787599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12169.178207] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12169.248587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12173.746183] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [12174.318020] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12179.428674] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12183.721926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12184.134698] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12184.182956] LustreError: Skipped 1 previous similar message [12184.194286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12185.204823] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [12189.269578] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [12190.691821] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12190.727301] LustreError: Skipped 3 previous similar messages [12197.562444] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12200.937106] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12200.965248] LustreError: Skipped 3 previous similar messages [12206.062458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12206.521957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12212.398415] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [12226.535417] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12226.544047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12226.578358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12227.041293] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12227.060680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12227.098767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12227.107089] Lustre: Skipped 1 previous similar message [12229.722949] Lustre: server umount lustre-OST0000 complete [12236.672290] Lustre: server umount lustre-MDT0000 complete [12237.280444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12237.287321] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12237.297903] Lustre: Skipped 1 previous similar message [12237.317916] LustreError: Skipped 3 previous similar messages [12241.048352] LustreError: 447296:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776211703 with bad export cookie 10821022603777569377 [12241.051385] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12241.057865] LustreError: 447296:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [12241.478615] Lustre: server umount lustre-MDT0001 complete [12255.898643] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12259.993416] Key type lgssc unregistered [12260.376143] LNet: 450582:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12260.388404] LNet: Removed LNI 192.168.202.119@tcp [12261.323234] Key type .llcrypt unregistered [12261.330747] Key type ._llcrypt unregistered [12291.534750] alg: No test for adler32 (adler32-zlib) [12292.300157] Key type ._llcrypt registered [12292.304869] Key type .llcrypt registered [12292.511811] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [12294.018886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12295.096343] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [12295.388873] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [12295.394873] LNet: Accept secure, port 988 [12297.119140] Key type lgssc registered [12298.330621] Lustre: Echo OBD driver; http://www.lustre.org/ [12308.628715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12325.407462] LustreError: 452042:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.119@tcp: failed processing log, type 1: rc = -5 [12356.127232] LustreError: 452042:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.119@tcp: failed processing log, type 4: rc = -110 [12386.975953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12392.629378] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [12401.456703] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12409.056557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12409.948774] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12410.060346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12410.133940] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [12415.465595] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12415.486573] LustreError: Skipped 1 previous similar message [12415.528974] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [12420.581040] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12425.310698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12425.768045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12431.093824] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [12439.444697] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12445.240455] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12452.326312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12452.364154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12454.184785] Lustre: server umount lustre-OST0000 complete [12458.351852] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12458.356672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12458.376117] Lustre: Skipped 1 previous similar message [12458.381682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12458.389781] Lustre: Skipped 1 previous similar message [12462.560447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12462.560967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12464.826310] Lustre: server umount lustre-MDT0000 complete [12467.682480] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12467.698400] LustreError: Skipped 1 previous similar message [12469.453871] LustreError: 454929:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776211931 with bad export cookie 15553805796484248648 [12469.455890] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12469.467860] LustreError: 454929:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [12469.777459] Lustre: server umount lustre-MDT0001 complete [12482.903933] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12486.061446] Key type lgssc unregistered [12486.352916] LNet: 455735:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12486.371755] LNet: Removed LNI 192.168.202.119@tcp [12487.429713] Key type .llcrypt unregistered [12487.432721] Key type ._llcrypt unregistered [12513.773564] alg: No test for adler32 (adler32-zlib) [12514.529658] Key type ._llcrypt registered [12514.532201] Key type .llcrypt registered [12514.700310] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [12515.771348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12516.549569] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [12516.738555] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [12516.742910] LNet: Accept secure, port 988 [12518.423103] Key type lgssc registered [12519.440357] Lustre: Echo OBD driver; http://www.lustre.org/ [12528.329307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12529.795484] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12529.863438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12534.541374] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [12535.280252] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12540.390508] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12543.657724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12544.130672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12549.287380] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [12557.076378] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12563.389620] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12570.666212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12570.976363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12575.983586] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [12580.235133] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [12584.019852] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12590.052034] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12590.063067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12590.088325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12590.788862] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12590.863473] Lustre: server umount lustre-MDT0000 complete [12593.819535] LustreError: 457198:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.19@tcp arrived at 1776212055 with bad export cookie 11225319008530578409 [12595.689277] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12595.705233] LustreError: Skipped 2 previous similar messages [12595.858239] LustreError: 457198:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776212057 with bad export cookie 11225319008530577513 [12595.873743] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12595.879151] LustreError: 457198:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [12599.791830] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12599.817517] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12599.832483] Lustre: Skipped 2 previous similar messages [12599.839834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12599.849902] Lustre: Skipped 2 previous similar messages [12602.238500] Lustre: server umount lustre-OST0000 complete [12607.989189] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12608.007901] LustreError: Skipped 5 previous similar messages [12611.341637] Lustre: server umount lustre-MDT0001 complete [12624.522747] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12627.332928] Key type lgssc unregistered [12627.577214] LNet: 461025:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12627.596696] LNet: Removed LNI 192.168.202.119@tcp [12628.484804] Key type .llcrypt unregistered [12628.486626] Key type ._llcrypt unregistered [12654.847355] alg: No test for adler32 (adler32-zlib) [12655.604761] Key type ._llcrypt registered [12655.608874] Key type .llcrypt registered [12655.739357] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [12657.233394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12657.795617] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [12658.026208] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [12658.031930] LNet: Accept secure, port 988 [12659.727419] Key type lgssc registered [12660.840440] Lustre: Echo OBD driver; http://www.lustre.org/ [12669.853592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12671.717932] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12671.811178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12677.084499] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [12677.088597] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12682.217709] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12686.980547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12687.338267] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12687.559264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12693.049325] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [12702.068535] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12708.305734] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12716.505948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12716.872190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12721.152750] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [12721.963506] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:161 [12727.890866] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12732.390308] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12732.412565] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12732.422831] Lustre: Skipped 1 previous similar message [12734.633625] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.19@tcp (stopping) [12737.515497] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12737.523876] Lustre: Skipped 1 previous similar message [12737.852082] Lustre: server umount lustre-MDT0001 complete [12739.721825] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12747.744172] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12747.745490] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12747.757876] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12747.757887] Lustre: Skipped 1 previous similar message [12747.761167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12747.778618] LustreError: Skipped 2 previous similar messages [12750.109258] Lustre: server umount lustre-OST0000 complete [12760.543262] Lustre: 465562:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776212216/real 1776212216] req@0000000045e0ce56 x1862493399956800/t0(0) o251->MGC192.168.202.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776212222 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [12760.876192] Lustre: server umount lustre-MDT0000 complete [12776.963673] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12780.774333] Key type lgssc unregistered [12781.140114] LNet: 466318:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12781.171119] LNet: Removed LNI 192.168.202.119@tcp [12782.148800] Key type .llcrypt unregistered [12782.151921] Key type ._llcrypt unregistered [12815.557212] alg: No test for adler32 (adler32-zlib) [12816.309412] Key type ._llcrypt registered [12816.318492] Key type .llcrypt registered [12816.491608] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [12829.733559] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [12830.914802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12831.446831] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [12831.628825] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [12831.634455] LNet: Accept secure, port 988 [12833.311428] Key type lgssc registered [12834.376171] Lustre: Echo OBD driver; http://www.lustre.org/ [12845.044268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12852.384836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12858.087759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12864.089765] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12874.717620] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [12886.107743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12894.151439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12894.262500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12895.578844] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [12895.626139] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [12895.789203] Lustre: lustre-MDT0000: new disk, initializing [12895.897123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12895.935345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12901.549580] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [12912.347962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12919.748427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12919.861531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12919.866863] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [12920.103857] Lustre: lustre-OST0000: new disk, initializing [12920.112788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [12920.124352] Lustre: Skipped 1 previous similar message [12920.194173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12924.482495] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [12935.368634] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12944.391065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12952.580581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12952.695636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12952.855896] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [12952.941955] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [12953.092741] Lustre: lustre-MDT0001: new disk, initializing [12953.166059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12953.175279] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12953.262797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12958.059959] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [12966.756966] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [12967.010920] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12971.567534] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [12971.866531] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12977.327285] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.19@tcp (stopping) [12980.192379] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12980.193576] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12980.201225] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12980.215658] Lustre: Skipped 1 previous similar message [12980.225380] Lustre: Skipped 1 previous similar message [12982.419155] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.19@tcp (stopping) [12982.702291] Lustre: server umount lustre-MDT0001 complete [12985.314893] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12990.435186] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12990.453048] LustreError: Skipped 2 previous similar messages [12995.553151] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12995.578133] LustreError: Skipped 1 previous similar message [13000.675655] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13000.682144] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13000.690414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13000.691293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13005.914380] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13006.114424] Lustre: server umount lustre-OST0000 complete [13025.247455] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [13031.903441] Lustre: 474733:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776212487/real 1776212487] req@000000002a7a9027 x1862493581364672/t0(0) o251->MGC192.168.202.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776212493 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [13032.177047] Lustre: server umount lustre-MDT0000 complete [13048.434500] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [13051.215516] Key type lgssc unregistered [13051.568235] LNet: 475492:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13051.582375] LNet: Removed LNI 192.168.202.119@tcp [13052.374433] Key type .llcrypt unregistered [13052.376472] Key type ._llcrypt unregistered [13100.559279] alg: No test for adler32 (adler32-zlib) [13101.315072] Key type ._llcrypt registered [13101.320566] Key type .llcrypt registered [13101.445720] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [13101.679975] Key type .llcrypt unregistered [13101.682518] Key type ._llcrypt unregistered [13117.848218] alg: No test for adler32 (adler32-zlib) [13118.622377] Key type ._llcrypt registered [13118.632909] Key type .llcrypt registered [13118.787712] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [13119.879631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13120.531362] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [13120.804977] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [13120.815486] LNet: Accept secure, port 988 [13122.495226] Key type lgssc registered [13123.503967] Lustre: Echo OBD driver; http://www.lustre.org/ [13132.496039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13138.823336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13143.952191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13150.442969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13150.528322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13151.934345] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [13151.970773] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [13152.055519] Lustre: lustre-MDT0000: new disk, initializing [13152.156660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13152.177094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13157.911357] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [13167.870096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13167.932507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13168.078964] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [13168.128127] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [13168.135871] Lustre: Skipped 1 previous similar message [13168.256751] Lustre: lustre-MDT0001: new disk, initializing [13168.366789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13168.422429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13168.436776] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13173.791601] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [13188.326969] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13196.660335] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13206.447509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13206.536078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13206.554843] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [13206.993630] Lustre: lustre-OST0000: new disk, initializing [13206.999393] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [13207.076363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13212.871754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13212.903544] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13214.937540] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [13225.391062] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13233.636388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13233.654783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13233.664589] Lustre: Skipped 1 previous similar message [13233.688673] Lustre: Skipped 2 previous similar messages [13238.115072] Lustre: server umount lustre-MDT0000 complete [13238.754485] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13238.771331] LustreError: Skipped 2 previous similar messages [13241.803305] LustreError: 479003:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776212703 with bad export cookie 14313094184880059009 [13241.826984] LustreError: 479003:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [13241.832184] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13242.295768] Lustre: server umount lustre-MDT0001 complete [13252.063561] Lustre: 482265:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776212708/real 1776212708] req@0000000094cb7880 x1862493884402112/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776212714 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [13252.476446] Lustre: server umount lustre-OST0000 complete [13264.533357] Lustre: DEBUG MARKER: == conf-sanity test 73: failnode to update from mountdata properly ========================================================== 20:25:25 (1776212725) [13265.871815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13272.700331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13273.039309] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13273.136863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13278.181643] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13278.403470] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [13283.310084] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13287.179849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13287.606096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13293.806732] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [13303.840989] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13309.626975] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13318.792978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13319.045657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13319.324395] Lustre: Found index 0 for lustre-OST0000, updating log [13319.487404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13324.617424] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5 to 0x0:33 [13326.284952] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [13336.605708] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13343.214603] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13343.217917] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13343.219690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13343.240957] Lustre: Skipped 2 previous similar messages [13343.941018] Lustre: server umount lustre-OST0000 complete [13348.328147] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13348.344593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13348.354376] Lustre: Skipped 2 previous similar messages [13353.444168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13353.455325] Lustre: Skipped 1 previous similar message [13358.568833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13358.574819] Lustre: Skipped 1 previous similar message [13362.658594] Lustre: server umount lustre-MDT0000 complete [13366.589727] LustreError: 483072:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776212828 with bad export cookie 14313094184880060346 [13366.610678] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13375.190419] Lustre: DEBUG MARKER: == conf-sanity test 75: The order of --index should be irrelevant ========================================================== 20:27:16 (1776212836) [13380.602621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13385.151384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13390.457130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13395.647547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13413.241543] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [13421.633861] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [13429.516359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13435.344536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13440.867230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13448.640159] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13456.750458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13456.840605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13457.086051] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [13457.128817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [13457.251582] Lustre: lustre-MDT0000: new disk, initializing [13457.358223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13457.380353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13461.302483] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [13470.780713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13470.871921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13470.960353] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [13470.986969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [13470.995321] Lustre: Skipped 1 previous similar message [13471.077476] Lustre: lustre-MDT0001: new disk, initializing [13471.205369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13471.223708] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13476.368037] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [13486.957978] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13494.118910] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13501.139620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13501.209755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13501.219698] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [13501.530535] Lustre: lustre-OST0000: new disk, initializing [13501.539961] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [13501.683484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13501.686395] Lustre: Skipped 1 previous similar message [13508.241634] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all 8 [13511.260852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13511.271332] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13520.582151] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13525.542340] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [13525.733567] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13530.615242] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [13530.820906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13537.249150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13537.256300] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13537.276842] Lustre: Skipped 1 previous similar message [13537.287541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13537.297064] Lustre: Skipped 3 previous similar messages [13546.975099] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [13547.125176] Lustre: server umount lustre-OST0000 complete [13547.127633] Lustre: Skipped 1 previous similar message [13552.610440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13552.628640] Lustre: Skipped 1 previous similar message [13557.729654] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13557.755315] LustreError: Skipped 1 previous similar message [13560.442230] LustreError: 490116:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776213022 with bad export cookie 14313094184880061410 [13560.445919] LustreError: 166-1: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13560.453898] LustreError: 490116:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [13591.090298] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [13600.183832] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [13613.502736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13624.263243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13632.385825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13642.692963] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13644.720728] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13411 sec ========= 20:31:45 (1776213105) [13692.428338] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [13695.684917] Key type lgssc unregistered [13696.175742] LNet: 498862:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13696.195235] LNet: Removed LNI 192.168.202.119@tcp [13697.540963] Key type .llcrypt unregistered [13697.543783] Key type ._llcrypt unregistered