[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 682152176 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.980 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2480MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002350] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.006962] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22982479f67, max_idle_ns: 440795221274 ns [ 0.007023] Calibrating delay loop (skipped) preset value.. 4799.96 BogoMIPS (lpj=2399980) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009142] LSM: Security Framework initializing [ 0.010070] Yama: becoming mindful. [ 0.011043] SELinux: Initializing. [ 0.012072] *** VALIDATE selinux *** [ 0.021558] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026729] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029084] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031289] *** VALIDATE tmpfs *** [ 0.033099] *** VALIDATE proc *** [ 0.034254] *** VALIDATE cgroup *** [ 0.035012] *** VALIDATE cgroup2 *** [ 0.036290] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037156] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039028] Spectre V2 : User space: Vulnerable [ 0.040008] Speculative Store Bypass: Vulnerable [ 0.043810] debug: unmapping init [mem 0xffffffffaa459000-0xffffffffaa460fff] [ 0.045176] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047054] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048023] ... version: 2 [ 0.049012] ... bit width: 48 [ 0.050015] ... generic registers: 4 [ 0.051013] ... value mask: 0000ffffffffffff [ 0.052016] ... max period: 00007fffffffffff [ 0.053183] ... fixed-purpose events: 3 [ 0.054232] ... event mask: 000000070000000f [ 0.055295] rcu: Hierarchical SRCU implementation. [ 0.057563] smp: Bringing up secondary CPUs ... [ 0.058792] x86: Booting SMP configuration: [ 0.059031] .... node #0, CPUs: #1 #2 #3 [ 0.067257] smp: Brought up 1 node, 4 CPUs [ 0.069016] smpboot: Max logical packages: 1 [ 0.070012] smpboot: Total of 4 processors activated (19199.84 BogoMIPS) [ 0.097112] node 0 deferred pages initialised in 25ms [ 0.102009] devtmpfs: initialized [ 0.103803] x86/mm: Memory block size: 128MB [ 0.107128] gcov: version magic: 0x41383552 [ 0.109028] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.112098] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.116446] pinctrl core: initialized pinctrl subsystem [ 0.119179] [ 0.121009] ************************************************************* [ 0.124015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.125012] ** ** [ 0.128016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.132016] ** ** [ 0.136013] ** This means that this kernel is built to expose internal ** [ 0.139015] ** IOMMU data structures, which may compromise security on ** [ 0.143016] ** your system. ** [ 0.146023] ** ** [ 0.151017] ** If you see this message and you are not debugging the ** [ 0.155057] ** kernel, report this immediately to your vendor! ** [ 0.159016] ** ** [ 0.163029] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168015] ************************************************************* [ 0.173162] NET: Registered protocol family 16 [ 0.175832] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.179064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.182071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.186360] cpuidle: using governor menu [ 0.188000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.191457] PCI: Using configuration type 1 for base access [ 0.194595] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.207196] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.210021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.214108] cryptd: max_cpu_qlen set to 1000 [ 0.217236] ACPI: Added _OSI(Module Device) [ 0.218014] ACPI: Added _OSI(Processor Device) [ 0.219111] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.221064] ACPI: Added _OSI(Processor Aggregator Device) [ 0.228310] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.236628] ACPI: Interpreter enabled [ 0.239074] ACPI: PM: (supports S0 S3 S4 S5) [ 0.241013] ACPI: Using IOAPIC for interrupt routing [ 0.243173] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.247613] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.260000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.263047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.266018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.270075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.277000] acpiphp: Slot [2] registered [ 0.279103] acpiphp: Slot [5] registered [ 0.281112] acpiphp: Slot [6] registered [ 0.284120] acpiphp: Slot [7] registered [ 0.286100] acpiphp: Slot [8] registered [ 0.288103] acpiphp: Slot [9] registered [ 0.289114] acpiphp: Slot [10] registered [ 0.291119] acpiphp: Slot [3] registered [ 0.293129] acpiphp: Slot [4] registered [ 0.295276] acpiphp: Slot [11] registered [ 0.298114] acpiphp: Slot [12] registered [ 0.300099] acpiphp: Slot [13] registered [ 0.302096] acpiphp: Slot [14] registered [ 0.304172] acpiphp: Slot [15] registered [ 0.306186] acpiphp: Slot [16] registered [ 0.308089] acpiphp: Slot [17] registered [ 0.310075] acpiphp: Slot [18] registered [ 0.312125] acpiphp: Slot [19] registered [ 0.314141] acpiphp: Slot [20] registered [ 0.316116] acpiphp: Slot [21] registered [ 0.318113] acpiphp: Slot [22] registered [ 0.319087] acpiphp: Slot [23] registered [ 0.321189] acpiphp: Slot [24] registered [ 0.324108] acpiphp: Slot [25] registered [ 0.325199] acpiphp: Slot [26] registered [ 0.328116] acpiphp: Slot [27] registered [ 0.330122] acpiphp: Slot [28] registered [ 0.332121] acpiphp: Slot [29] registered [ 0.334378] acpiphp: Slot [30] registered [ 0.337116] acpiphp: Slot [31] registered [ 0.340057] PCI host bridge to bus 0000:00 [ 0.342019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.345025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.349025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.353033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.357028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.360028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.363173] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.367307] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.372125] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.386885] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.393055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.396020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.400072] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.403022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.407596] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.411968] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.415049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.418953] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.426016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.442018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.448016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.453000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.463029] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.471024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.487027] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.498578] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.509022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.520017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.545019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.565555] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.578016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.593022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.643022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.662902] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.687018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.705017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.745017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.767848] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.781016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.793014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.831018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.852974] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.866018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.884023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.939028] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.970876] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.974456] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.977512] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.980384] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.983233] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.989096] iommu: Default domain type: Passthrough [ 0.991453] SCSI subsystem initialized [ 0.993166] ACPI: bus type USB registered [ 0.995087] usbcore: registered new interface driver usbfs [ 0.997151] usbcore: registered new interface driver hub [ 0.999083] usbcore: registered new device driver usb [ 1.001160] pps_core: LinuxPPS API ver. 1 registered [ 1.004011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.008208] PTP clock support registered [ 1.010285] EDAC MC: Ver: 3.0.0 [ 1.013259] PCI: Using ACPI for IRQ routing [ 1.016260] NetLabel: Initializing [ 1.018010] NetLabel: domain hash size = 128 [ 1.020010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.023135] NetLabel: unlabeled traffic allowed by default [ 1.026095] vgaarb: loaded [ 1.029287] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.031015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.041893] clocksource: Switched to clocksource kvm-clock [ 1.181840] VFS: Disk quotas dquot_6.6.0 [ 1.183324] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.186489] *** VALIDATE ramfs *** [ 1.187808] *** VALIDATE hugetlbfs *** [ 1.189392] pnp: PnP ACPI init [ 1.192743] pnp: PnP ACPI: found 6 devices [ 1.210274] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.214400] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.217070] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.219511] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.222420] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.225595] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.229517] NET: Registered protocol family 2 [ 1.232731] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.238404] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.242254] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.248577] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.251989] TCP: Hash tables configured (established 65536 bind 65536) [ 1.254979] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.258256] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.261544] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.264670] NET: Registered protocol family 1 [ 1.267450] RPC: Registered named UNIX socket transport module. [ 1.269681] RPC: Registered udp transport module. [ 1.271608] RPC: Registered tcp transport module. [ 1.273496] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.276068] NET: Registered protocol family 44 [ 1.277686] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.280217] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.282323] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.284448] PCI: CLS 0 bytes, default 64 [ 1.286123] Unpacking initramfs... [ 3.011570] debug: unmapping init [mem 0xffff9116bcc54000-0xffff9116bffbffff] [ 3.016432] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.019066] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 3.022673] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22982479f67, max_idle_ns: 440795221274 ns [ 3.578259] Initialise system trusted keyrings [ 3.580302] Key type blacklist registered [ 3.582735] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.593019] zbud: loaded [ 3.596322] *** VALIDATE nfs *** [ 3.597391] *** VALIDATE nfs4 *** [ 3.598881] pstore: using deflate compression [ 3.601904] Platform Keyring initialized [ 3.717903] NET: Registered protocol family 38 [ 3.719984] Key type asymmetric registered [ 3.722231] Asymmetric key parser 'x509' registered [ 3.725528] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.729838] io scheduler mq-deadline registered [ 3.731496] io scheduler kyber registered [ 3.734169] io scheduler bfq registered [ 3.736119] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.738630] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.741645] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.744692] ACPI: Power Button [PWRF] [ 3.751447] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.761196] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.781607] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.792712] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.816741] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.847305] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.877035] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.883579] Non-volatile memory driver v1.3 [ 3.885615] Linux agpgart interface v0.103 [ 3.925265] virtio_blk virtio1: [vda] 134776 512-byte logical blocks (69.0 MB/65.8 MiB) [ 3.930644] vda: detected capacity change from 0 to 69005312 [ 3.953333] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.956609] vdb: detected capacity change from 0 to 1073741824 [ 3.979022] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.982218] vdc: detected capacity change from 0 to 2621440000 [ 4.001273] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.004058] vdd: detected capacity change from 0 to 2621440000 [ 4.052208] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.056465] vde: detected capacity change from 0 to 4294967296 [ 4.085617] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.088633] vdf: detected capacity change from 0 to 4294967296 [ 4.099624] libphy: Fixed MDIO Bus: probed [ 4.107563] usbcore: registered new interface driver usbserial_generic [ 4.111314] usbserial: USB Serial support registered for generic [ 4.114105] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.119722] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.122283] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.125106] mousedev: PS/2 mouse device common for all mice [ 4.128140] rtc_cmos 00:05: RTC can wake from S4 [ 4.131653] rtc_cmos 00:05: registered as rtc0 [ 4.131730] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.133117] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.140685] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.141854] intel_pstate: CPU model not supported [ 4.148668] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.153813] hid: raw HID events driver (C) Jiri Kosina [ 4.156569] usbcore: registered new interface driver usbhid [ 4.158630] usbhid: USB HID core driver [ 4.160430] drop_monitor: Initializing network drop monitor service [ 4.163028] Initializing XFRM netlink socket [ 4.165321] NET: Registered protocol family 10 [ 4.168537] Segment Routing with IPv6 [ 4.170123] NET: Registered protocol family 17 [ 4.172213] mpls_gso: MPLS GSO support [ 4.178689] RAS: Correctable Errors collector initialized. [ 4.181216] AVX version of gcm_enc/dec engaged. [ 4.183207] AES CTR mode by8 optimization enabled [ 4.274737] sched_clock: Marking stable (4274708997, 0)->(5425165795, -1150456798) [ 4.279548] registered taskstats version 1 [ 4.282243] Loading compiled-in X.509 certificates [ 4.285338] zswap: loaded using pool lzo/zbud [ 4.317599] Key type big_key registered [ 4.334420] Key type encrypted registered [ 4.336323] ima: No TPM chip found, activating TPM-bypass! [ 4.338275] ima: Allocated hash algorithm: sha1 [ 4.340346] ima: No architecture policies found [ 4.342250] evm: Initialising EVM extended attributes: [ 4.343965] evm: security.selinux [ 4.345434] evm: security.ima [ 4.346689] evm: security.capability [ 4.348563] evm: HMAC attrs: 0x1 [ 4.351579] rtc_cmos 00:05: setting system clock to 2026-05-25 07:09:49 UTC (1779692989) [ 4.359610] debug: unmapping init [mem 0xffffffffab403000-0xffffffffab5fffff] [ 4.363492] debug: unmapping init [mem 0xffffffffaa182000-0xffffffffaa458fff] [ 4.373162] Write protecting the kernel read-only data: 28672k [ 4.377229] debug: unmapping init [mem 0xffffffffa8803000-0xffffffffa89fffff] [ 4.380788] debug: unmapping init [mem 0xffffffffa9114000-0xffffffffa91fffff] [ 4.443628] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.451392] systemd[1]: Detected virtualization kvm. [ 4.453185] systemd[1]: Detected architecture x86-64. [ 4.454941] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.485267] systemd[1]: No hostname configured. [ 4.486755] systemd[1]: Set hostname to . [ 4.488699] random: systemd: uninitialized urandom read (16 bytes read) [ 4.491248] systemd[1]: Initializing machine ID from random generator. [ 4.706616] random: systemd: uninitialized urandom read (16 bytes read) [ 4.709704] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.713758] random: systemd: uninitialized urandom read (16 bytes read) [ 4.716689] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.729548] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.761450] device-mapper: uevent: version 1.0.3 [ 5.765522] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ 6.781714] random: fast init done [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.841381] virtio_net virtio0 ens2: renamed from eth0 [ 7.631058] scsi host0: ata_piix [ 7.649332] scsi host1: ata_piix [ 7.651962] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.655286] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.865806] random: crng init done [ 11.880454] random: 7 urandom warning(s) missed due to ratelimiting [ 15.152988] dracut-initqueue[582]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 19.335525] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ 22.007594] hrtimer: interrupt took 5270329 ns Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 25.178201] printk: systemd: 17 output lines suppressed due to ratelimiting [ 27.483309] SELinux: Disabled at runtime. [ 28.042037] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 28.127602] systemd[1]: Detected virtualization kvm. [ 28.142120] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 32.866412] systemd[1]: initrd-switch-root.service: Succeeded. [ 32.877955] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 32.897091] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 32.902520] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 32.913073] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 32.929488] systemd[1]: Starting Journal Service... Starting Journal Service... [ 32.956901] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ 33.086183] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ 33.121966] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ 33.185035] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ 33.222784] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 33.293757] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 33.322867] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 33.428992] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ 33.465244] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 33.535150] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 33.537102] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ 33.587084] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 38.197309] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 40.303917] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 40.349836] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 42.395658] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 42.697175] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…only root support (11s / no limit) [** ] A start job is running for Configur…only root support (12s / no limit) [*** ] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (14s / no limit) [ **] A start job is running for Configur…only root support (15s / no limit) [ *] A start job is running for Configur…only root support (15s / no limit) [ **] A start job is running for Configur…only root support (16s / no limit) [ ***] A start job is running for Configur…only root support (17s / no limit) [ *** ] A start job is running for Configur…only root support (17s / no limit) [ *** ] A start job is running for Configur…only root support (18s / no limit) [*** ] A start job is running for Configur…only root support (18s / no limit) [** ] A start job is running for Configur…only root support (19s / no limit) [* ] A start job is running for Configur…only root support (20s / no limit) [** ] A start job is running for Configur…only root support (20s / no limit)[ 54.061825] Key type dns_resolver registered [*** ] A start job is running for Configur…only root support (21s / no limit) [ *** ] A start job is running for Configur…only root support (22s / no limit)[ 55.001176] NFS: Registering the id_resolver key type [ 55.013815] Key type id_resolver registered [ 55.022505] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (22s / no limit) [ ***] A start job is running for Configur…only root support (23s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg157-server login: [ 132.594819] libcfs: loading out-of-tree module taints kernel. [ 132.630655] Key type ._llcrypt registered [ 132.633857] Key type .llcrypt registered [ 132.768276] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 158.597619] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 161.061744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 161.077282] alg: No test for adler32 (adler32-zlib) [ 162.799826] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 164.223201] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 166.024206] Key type lgssc registered [ 168.246877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 193.504536] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 250.947408] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 267.020989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 267.054522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 268.474982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 268.519162] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 268.685565] Lustre: lustre-MDT0000: new disk, initializing [ 268.851757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 268.934646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 273.962221] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 291.212191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 291.488468] Lustre: 6528:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 291.591369] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 291.607284] Lustre: Skipped 1 previous similar message [ 291.847618] Lustre: lustre-MDT0001: new disk, initializing [ 291.949402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 292.017365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 292.043390] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 297.762447] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 303.971788] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 316.465909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 316.858672] Lustre: lustre-OST0000: new disk, initializing [ 316.864676] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 316.879743] Lustre: 8434:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 316.979631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 325.733654] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 326.213376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 326.232243] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 326.341781] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 348.047764] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 348.302852] Lustre: lustre-OST0001: new disk, initializing [ 348.311357] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 348.319155] Lustre: 9493:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 348.415833] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 356.864750] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 357.433376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 357.449765] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 357.519906] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 373.000195] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 382.198446] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 392.616873] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing check_logdir /tmp/testlogs/ [ 399.974728] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing yml_node [ 408.012391] Lustre: DEBUG MARKER: Client: 2.17.53.24 [ 412.007655] Lustre: DEBUG MARKER: MDS: 2.17.53.24 [ 415.446552] Lustre: DEBUG MARKER: OSS: 2.17.53.24 [ 418.152825] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon May 25 03:16:40 EDT 2026 [ 447.777739] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 450.693532] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 470.498105] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 470.508196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 470.529816] Lustre: Skipped 1 previous similar message [ 470.561177] Lustre: Skipped 3 previous similar messages [ 475.618756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 475.633872] Lustre: Skipped 3 previous similar messages [ 477.314737] Lustre: server umount lustre-MDT0000 complete [ 482.676929] LustreError: 6521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779693467 with bad export cookie 16994793289717503077 [ 482.689890] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 482.705341] LustreError: 6521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 485.866698] LustreError: 7672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 485.875798] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 485.888832] LustreError: 7672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 485.888868] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 485.942693] Lustre: Skipped 3 previous similar messages [ 487.905954] LustreError: 12028:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 487.906265] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 487.946204] LustreError: 12028:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 487.971417] Lustre: Skipped 2 previous similar messages [ 489.443686] Lustre: server umount lustre-MDT0001 complete [ 495.512984] Lustre: server umount lustre-OST0000 complete [ 502.018450] Lustre: server umount lustre-OST0001 complete [ 512.947749] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 529.446473] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 584.061282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 584.604803] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 584.668329] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 584.851690] Lustre: lustre-MDT0000: new disk, initializing [ 585.001820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 585.027429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 592.422256] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 608.850700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 609.061982] Lustre: 15039:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 609.195518] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 609.199161] Lustre: Skipped 1 previous similar message [ 609.598846] Lustre: lustre-MDT0001: new disk, initializing [ 609.810671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 609.918346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 609.958739] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 616.548347] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 628.535695] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 635.210866] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 646.885575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 647.447357] Lustre: lustre-OST0000: new disk, initializing [ 647.457442] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 647.468432] Lustre: 16784:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 647.676935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 648.752452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 648.795989] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 649.035232] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 657.195307] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 669.976652] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 677.944768] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 678.313995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 686.169849] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 686.560610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 690.149966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 690.171477] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 690.172267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 690.174920] Lustre: Skipped 1 previous similar message [ 700.393450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 700.409116] Lustre: Skipped 4 previous similar messages [ 703.983846] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 704.461885] Lustre: server umount lustre-OST0000 complete [ 710.628285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 716.365096] Lustre: server umount lustre-MDT0000 complete [ 720.866525] LustreError: 16430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 720.890776] LustreError: 16430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 721.400157] LustreError: 15031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779693706 with bad export cookie 16994793289717504610 [ 721.405492] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 721.415891] LustreError: 15031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 721.976585] Lustre: server umount lustre-MDT0001 complete [ 728.059560] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 03:21:50 (1779693710) [ 731.673542] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 734.156848] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 03:21:56 (1779693716) [ 743.937039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 744.655879] LustreError: 19051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 744.811880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 750.054116] LustreError: 19052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 750.854350] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 759.269750] LustreError: 19051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 759.304267] LustreError: 19051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 762.405904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 769.747793] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 778.631197] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 785.014818] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 796.294870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 805.567368] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 814.231333] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 827.361654] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 827.379493] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 827.407197] Lustre: Skipped 1 previous similar message [ 827.428000] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 827.440080] Lustre: Skipped 4 previous similar messages [ 829.420927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 831.054241] Lustre: server umount lustre-OST0000 complete [ 839.660230] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 839.683866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 839.685629] Lustre: Skipped 2 previous similar messages [ 843.083568] Lustre: server umount lustre-MDT0000 complete [ 844.796703] LustreError: 19051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 844.849696] LustreError: 19051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 848.655144] LustreError: 19034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779693833 with bad export cookie 16994793289717505492 [ 848.667405] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 848.678704] LustreError: 19034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 849.263794] Lustre: server umount lustre-MDT0001 complete [ 866.398819] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 870.549836] Key type lgssc unregistered [ 871.059901] LNet: 22233:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.071940] LNetError: 22233:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 872.102239] LNet: Removed LNI 192.168.201.157@tcp [ 873.667157] Key type .llcrypt unregistered [ 873.671264] Key type ._llcrypt unregistered [ 911.728305] Key type ._llcrypt registered [ 911.730984] Key type .llcrypt registered [ 911.874427] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 914.677581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 914.738829] alg: No test for adler32 (adler32-zlib) [ 916.087639] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 916.552033] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 918.332114] Key type lgssc registered [ 920.198867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 931.957065] loop: module loaded [ 932.092946] loop0: detected capacity change from 0 to 204800000 [ 932.426732] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 932.451354] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 933.019585] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 941.556352] loop0: detected capacity change from 0 to 204800000 [ 941.856140] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 941.882723] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 941.901829] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 948.701621] loop0: detected capacity change from 0 to 204800000 [ 956.828768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 956.869989] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 958.269149] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 958.334598] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 958.416335] Lustre: lustre-MDT0000: new disk, initializing [ 958.610851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 958.692490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 965.572228] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 978.897844] loop1: detected capacity change from 0 to 204800000 [ 987.534919] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 988.064982] Lustre: lustre-OST07c6: new disk, initializing [ 988.073050] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 988.085433] Lustre: Skipped 1 previous similar message [ 988.089955] Lustre: 25393:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 988.257765] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 996.459209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 996.469628] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 996.511595] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 996.997649] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1005.052639] Lustre: 24394:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 1005.070838] Lustre: 24394:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 1005.079502] Lustre: 24394:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 1005.091518] Lustre: 24394:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1005.100131] Lustre: 24394:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 1005.108129] Lustre: 24394:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1023.572725] Lustre: 24396:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 1023.590546] Lustre: 24396:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 1023.610563] Lustre: 24396:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1023.639156] Lustre: 24396:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 1023.654949] Lustre: 24396:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 1023.670517] Lustre: 24396:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1030.248675] Lustre: 25400:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 1030.259553] Lustre: 25400:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 1030.267524] Lustre: 25400:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 1030.286643] Lustre: 25400:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1030.307322] Lustre: 25400:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1030.340870] Lustre: 25400:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1030.367049] Lustre: 25400:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 1030.381974] Lustre: 25400:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1030.393454] Lustre: 25400:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 1030.406201] Lustre: 25400:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1030.416628] Lustre: 25400:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1030.428995] Lustre: 25400:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1033.805091] Lustre: 24395:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 1033.829745] Lustre: 24395:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 1033.845382] Lustre: 24395:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1033.856691] Lustre: 24395:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1033.881930] Lustre: 24395:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1033.899264] Lustre: 24395:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1033.917962] Lustre: 24395:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 1033.923931] Lustre: 24395:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1033.928872] Lustre: 24395:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 1033.934453] Lustre: 24395:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1033.940666] Lustre: 24395:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1033.945485] Lustre: 24395:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1043.393044] Lustre: 25400:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 1043.409491] Lustre: 25400:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 1043.417377] Lustre: 25400:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 1043.442185] Lustre: 25400:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1043.460183] Lustre: 25400:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1043.491331] Lustre: 25400:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1043.508406] Lustre: 25400:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 1043.518146] Lustre: 25400:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1043.543610] Lustre: 25400:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 1043.559223] Lustre: 25400:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1043.565291] Lustre: 25400:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1043.572808] Lustre: 25400:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1056.552786] Lustre: 24394:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 1056.562249] Lustre: 24394:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 1056.569557] Lustre: 24394:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1056.574383] Lustre: 24394:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1056.579767] Lustre: 24394:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1056.585941] Lustre: 24394:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1056.591267] Lustre: 24394:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 1056.603330] Lustre: 24394:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1056.612248] Lustre: 24394:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 1056.620222] Lustre: 24394:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1056.632727] Lustre: 24394:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1056.639189] Lustre: 24394:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1108.974624] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1108.991316] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1109.028720] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 1113.470773] Lustre: server umount lustre-OST07c6 complete [ 1127.581535] Lustre: server umount lustre-MDT0000 complete [ 1151.600531] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 1155.806046] Key type lgssc unregistered [ 1156.273208] LNet: 27920:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1156.280749] LNetError: 27920:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1157.352252] LNet: Removed LNI 192.168.201.157@tcp [ 1158.741243] Key type .llcrypt unregistered [ 1158.748912] Key type ._llcrypt unregistered [ 1187.279114] Key type ._llcrypt registered [ 1187.293446] Key type .llcrypt registered [ 1187.540953] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 1189.420521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1189.493063] alg: No test for adler32 (adler32-zlib) [ 1190.853461] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 1191.419808] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 1193.195700] Key type lgssc registered [ 1195.229919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1207.900295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1207.969683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1209.900117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1216.125737] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1219.045533] LustreError: 29176:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1219.074468] LustreError: 29176:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1224.174802] LustreError: 29175:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1228.105584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1228.502891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1234.375048] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1243.526750] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1249.849489] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1259.409139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1259.854439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1268.809409] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1277.680404] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1280.389391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1306.097329] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1306.117340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1306.148940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1306.161120] Lustre: Skipped 1 previous similar message [ 1306.196150] Lustre: Skipped 1 previous similar message [ 1311.217597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1311.238644] Lustre: Skipped 1 previous similar message [ 1316.323816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1316.333986] Lustre: Skipped 1 previous similar message [ 1319.392626] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1319.637641] Lustre: server umount lustre-OST0000 complete [ 1326.050853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1326.072213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1326.101418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1331.687383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1331.699930] Lustre: Skipped 2 previous similar messages [ 1340.385567] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1340.631921] Lustre: server umount lustre-MDT0000 complete [ 1341.926322] LustreError: 29887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1341.953046] LustreError: 29887:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1345.638064] LustreError: 29156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779694330 with bad export cookie 9418441341835370612 [ 1345.653421] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1345.654322] LustreError: 29156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1346.014612] Lustre: server umount lustre-MDT0001 complete [ 1362.716272] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 1366.808355] Key type lgssc unregistered [ 1367.203652] LNet: 32580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1367.219848] LNetError: 32580:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1368.296223] LNet: Removed LNI 192.168.201.157@tcp [ 1369.652912] Key type .llcrypt unregistered [ 1369.658670] Key type ._llcrypt unregistered [ 1396.232845] Key type ._llcrypt registered [ 1396.246780] Key type .llcrypt registered [ 1396.358702] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 1397.659913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1398.023750] alg: No test for adler32 (adler32-zlib) [ 1399.141836] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 1399.422437] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 1401.153545] Key type lgssc registered [ 1402.988829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1415.493799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1415.516053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1417.696594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1424.226076] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1426.926279] LustreError: 33838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1426.962970] LustreError: 33838:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1432.037419] LustreError: 33839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1435.866687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1436.476148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1442.612292] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1451.149799] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1456.658484] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1466.882965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1467.254523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1475.823162] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1484.101286] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1487.856111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1506.411480] Lustre: server umount lustre-OST0000 complete [ 1513.447045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1513.464461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1513.473127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1513.508090] Lustre: Skipped 1 previous similar message [ 1518.564474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1518.580383] Lustre: Skipped 2 previous similar messages [ 1522.768606] Lustre: server umount lustre-MDT0000 complete [ 1523.681427] Lustre: 33151:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779694492/real 1779694492] req@0000000026b73566 x1866144818429568/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1779694508 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1523.688426] LustreError: 33839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1523.731748] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1523.791403] LustreError: 33839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1524.704350] Lustre: 33149:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779694493/real 1779694493] req@00000000e1f2e551 x1866144818430336/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779694509 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1528.584716] LustreError: 33821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779694513 with bad export cookie 231832747193346004 [ 1528.613827] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1529.292158] Lustre: server umount lustre-MDT0001 complete [ 1546.564728] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 1550.785268] Key type lgssc unregistered [ 1551.194407] LNet: 37040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1551.204784] LNetError: 37040:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1551.218843] LNet: Removed LNI 192.168.201.157@tcp [ 1552.232744] Key type .llcrypt unregistered [ 1552.237145] Key type ._llcrypt unregistered [ 1584.425298] Key type ._llcrypt registered [ 1584.427387] Key type .llcrypt registered [ 1584.589730] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 1586.206678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1586.498359] alg: No test for adler32 (adler32-zlib) [ 1587.642666] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 1587.919395] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 1589.608737] Key type lgssc registered [ 1591.617097] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1603.451495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1603.502281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1605.605646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1612.527507] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1615.844711] LustreError: 38443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1615.873239] LustreError: 38443:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1620.973665] LustreError: 38444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1625.779632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1626.268769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1633.292481] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1642.828232] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1649.153635] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1659.795087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1660.623590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1670.074695] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1675.017163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1678.636372] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1690.598877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1690.611230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1690.615506] Lustre: Skipped 1 previous similar message [ 1690.643357] Lustre: Skipped 2 previous similar messages [ 1692.107583] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.57@tcp (stopping) [ 1694.101815] Lustre: server umount lustre-MDT0000 complete [ 1695.714588] LustreError: 38444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1695.764168] LustreError: 38444:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1699.566301] LustreError: 39162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779694684 with bad export cookie 17216020358939965586 [ 1699.569809] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1699.601272] LustreError: 39162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1700.834462] LustreError: 38440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1700.835443] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1700.867302] LustreError: 38440:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1700.899730] Lustre: Skipped 1 previous similar message [ 1700.913921] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1704.870858] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1704.871745] LustreError: 40217:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1704.892361] Lustre: Skipped 1 previous similar message [ 1704.923277] LustreError: 40217:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1706.298955] Lustre: server umount lustre-MDT0001 complete [ 1723.721691] Lustre: server umount lustre-OST0000 complete [ 1747.378666] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 1750.948165] Key type lgssc unregistered [ 1751.439955] LNet: 41955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1751.455255] LNetError: 41955:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1751.483260] LNet: Removed LNI 192.168.201.157@tcp [ 1752.718532] Key type .llcrypt unregistered [ 1752.721902] Key type ._llcrypt unregistered [ 1785.740300] Key type ._llcrypt registered [ 1785.742221] Key type .llcrypt registered [ 1785.855514] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 1786.913348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1786.932872] alg: No test for adler32 (adler32-zlib) [ 1788.496830] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 1788.975565] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 1790.718098] Key type lgssc registered [ 1792.336729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1801.771254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1801.799293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1803.452693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1808.612729] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1812.968849] LustreError: 43358:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1812.991583] LustreError: 43358:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1818.084371] LustreError: 43359:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1819.167298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1819.631182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1824.606477] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1832.242679] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1836.986100] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1845.965114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1846.424479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1848.528093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1854.043266] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1861.644696] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1873.880911] Lustre: lustre-OST0000: Not available for connect from 192.168.201.57@tcp (stopping) [ 1874.405172] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1874.414901] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1874.433643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1875.431237] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1875.446336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1875.447303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1875.489534] Lustre: Skipped 1 previous similar message [ 1877.592677] Lustre: server umount lustre-OST0000 complete [ 1897.385504] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1897.410843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1897.414906] Lustre: Skipped 1 previous similar message [ 1897.432820] Lustre: Skipped 1 previous similar message [ 1903.795769] Lustre: server umount lustre-MDT0000 complete [ 1904.610795] LustreError: 43355:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1904.648220] LustreError: 43355:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1908.191196] LustreError: 43340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779694893 with bad export cookie 11957110562736087136 [ 1908.196133] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1908.220175] LustreError: 43340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1914.697620] Lustre: server umount lustre-MDT0001 complete [ 1929.814633] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 1933.386956] Key type lgssc unregistered [ 1933.759539] LNet: 46713:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1933.772208] LNetError: 46713:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1933.797880] LNet: Removed LNI 192.168.201.157@tcp [ 1935.097309] Key type .llcrypt unregistered [ 1935.101405] Key type ._llcrypt unregistered [ 1968.056955] Key type ._llcrypt registered [ 1968.061904] Key type .llcrypt registered [ 1968.195989] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 1970.224392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1970.241865] alg: No test for adler32 (adler32-zlib) [ 1971.545750] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 1971.945833] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 1973.752335] Key type lgssc registered [ 1975.546105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1985.354850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1985.378384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1987.115742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1993.223639] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1997.283712] LustreError: 48114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1997.305140] LustreError: 48114:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2002.404776] LustreError: 48113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2005.348504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2005.896348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2012.027382] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2020.429703] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2026.143825] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2035.385054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2036.013616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2045.129532] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2052.162854] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2057.526575] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2064.106610] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2068.434702] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 2081.372425] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-05-25 3h44m22s [ 2083.586724] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 2087.906260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2087.916923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2087.954549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2089.002334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2089.014630] Lustre: Skipped 4 previous similar messages [ 2090.966428] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.57@tcp (stopping) [ 2090.972121] Lustre: Skipped 1 previous similar message [ 2092.882430] Lustre: server umount lustre-MDT0000 complete [ 2093.103603] LustreError: 48826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2093.134185] LustreError: 48826:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2097.075053] LustreError: 48109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2097.206512] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 2103.192273] LustreError: 48108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2103.210312] LustreError: 48108:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2106.532111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2106.805130] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2107.212127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2107.338503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2108.215495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2112.972757] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2118.023293] Lustre: DEBUG MARKER: Wait for df (36147) ... [ 2120.615258] Lustre: DEBUG MARKER: done [ 2127.331831] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2127.338847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2127.345951] Lustre: Skipped 2 previous similar messages [ 2127.349172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2132.726379] Lustre: server umount lustre-OST0000 complete [ 2139.108736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2139.109608] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2139.117641] LustreError: Skipped 1 previous similar message [ 2139.119151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2139.155254] Lustre: Skipped 2 previous similar messages [ 2139.165448] Lustre: Skipped 3 previous similar messages [ 2144.125882] Lustre: server umount lustre-MDT0000 complete [ 2146.789831] LustreError: 48826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2146.815924] LustreError: 48826:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2149.625841] LustreError: 48833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779695134 with bad export cookie 8715046265149453719 [ 2149.632061] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2149.657141] LustreError: 48833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2150.354525] Lustre: server umount lustre-MDT0001 complete [ 2164.977249] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 2169.096685] Key type lgssc unregistered [ 2169.561270] LNet: 52690:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2169.587760] LNetError: 52690:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2169.621625] LNet: Removed LNI 192.168.201.157@tcp [ 2170.723187] Key type .llcrypt unregistered [ 2170.735829] Key type ._llcrypt unregistered [ 2222.898386] Key type ._llcrypt registered [ 2222.900048] Key type .llcrypt registered [ 2223.183800] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 2225.517704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2225.559467] alg: No test for adler32 (adler32-zlib) [ 2226.811639] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 2227.176986] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 2229.008577] Key type lgssc registered [ 2232.105098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2243.157297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2243.190312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2244.836261] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2244.909932] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2245.244512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2252.365968] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2263.963581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2264.125922] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2264.167577] Lustre: 54886:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2264.420042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2270.401595] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2278.742676] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2285.443413] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2296.936894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2297.181124] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2297.442488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2303.851275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2307.794515] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2316.062436] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2325.222262] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-05-25 3h48m27s [ 2327.435317] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2336.556248] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2340.100120] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2340.341893] LustreError: 56529:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2360.804153] Lustre: lustre-MDT0000: Client 3000f1ac-912a-4882-adc1-8d070ffdf263 (at 192.168.201.57@tcp) reconnecting [ 2380.400103] LustreError: 56529:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2383.762946] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2387.025061] Lustre: DEBUG MARKER: done [ 2396.130641] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2396.148695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2396.161212] LustreError: Skipped 1 previous similar message [ 2396.185439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2401.257050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2401.272757] Lustre: Skipped 1 previous similar message [ 2401.749500] Lustre: server umount lustre-OST0000 complete [ 2411.503620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2411.550124] Lustre: Skipped 1 previous similar message [ 2411.576489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2411.593578] Lustre: Skipped 1 previous similar message [ 2419.184861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2419.191328] Lustre: Skipped 2 previous similar messages [ 2424.292565] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2424.654674] Lustre: server umount lustre-MDT0000 complete [ 2429.564194] LustreError: 54877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779695414 with bad export cookie 14048452836334857183 [ 2429.581161] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2429.581864] LustreError: 54877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2430.066595] Lustre: server umount lustre-MDT0001 complete [ 2443.643261] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 2447.166860] Key type lgssc unregistered [ 2447.568205] LNet: 58874:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2447.579891] LNetError: 58874:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2447.608995] LNet: Removed LNI 192.168.201.157@tcp [ 2448.696452] Key type .llcrypt unregistered [ 2448.699395] Key type ._llcrypt unregistered [ 2510.092515] Key type ._llcrypt registered [ 2510.096466] Key type .llcrypt registered [ 2510.243627] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 2511.897277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2511.958148] alg: No test for adler32 (adler32-zlib) [ 2513.123837] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 2513.393762] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 2515.120330] Key type lgssc registered [ 2516.900784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2521.050320] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2521.072500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2522.362484] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2522.401664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2522.743550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2524.168084] Lustre: Failing over lustre-MDT0000 [ 2524.462672] Lustre: server umount lustre-MDT0000 complete [ 2532.974309] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 03:51:55 (1779695515) [ 2541.893153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2542.433788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2549.038295] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2560.007269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2560.100966] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2560.139446] Lustre: 61660:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2560.356905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2566.567860] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2575.203729] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2581.547968] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2590.448956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2590.623587] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2590.827670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2598.040473] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2599.239537] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2605.771457] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2612.516566] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2623.983582] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2623.989563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2624.011122] Lustre: Skipped 2 previous similar messages [ 2624.031713] Lustre: Skipped 2 previous similar messages [ 2628.866744] Lustre: server umount lustre-MDT0000 complete [ 2634.220607] LustreError: 61667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2634.257468] LustreError: 61667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2634.469027] LustreError: 61651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779695619 with bad export cookie 3389497307541193159 [ 2634.478178] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2634.493118] LustreError: 61651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2634.928688] Lustre: server umount lustre-MDT0001 complete [ 2640.148820] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2646.705222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2655.713453] Lustre: 60583:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779695624/real 1779695624] req@000000003eab4585 x1866145986555392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779695640 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2655.752987] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2658.413083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2659.553249] LustreError: 65077:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2659.566208] LustreError: 65077:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000f72c41eb x1866145986555520/t0(0) o250->MGC192.168.201.157@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1779695644 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2659.603132] LustreError: 65077:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 2659.875919] LustreError: 60580:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000035707a8e x1866145986556160/t0(0) o250->MGC192.168.201.157@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2660.409845] LustreError: 63281:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2660.577614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2660.766206] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2661.537293] Lustre: 60584:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779695629/real 1779695629] req@00000000f1ffacc0 x1866145986555648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779695645 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2664.678479] LustreError: 63282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2664.689133] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2665.696512] Lustre: 60581:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779695634/real 1779695634] req@0000000010676abd x1866145986555776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779695650 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2665.932541] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2669.857449] Lustre: 60582:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779695639/real 1779695639] req@00000000adecbdfe x1866145986555904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779695655 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2671.081635] LustreError: 65091:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2671.129275] LustreError: 65091:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2676.206327] LustreError: 65089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2676.243359] LustreError: 65089:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2677.039959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2677.457708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2682.863768] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2683.004859] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2690.619164] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2696.747458] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2708.450586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2708.456082] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2708.463884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2708.482546] Lustre: Skipped 2 previous similar messages [ 2713.571939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2713.588152] Lustre: Skipped 4 previous similar messages [ 2718.700307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2718.720288] Lustre: Skipped 2 previous similar messages [ 2720.224287] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2720.485470] Lustre: server umount lustre-MDT0000 complete [ 2723.823692] LustreError: 65091:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2723.853563] LustreError: 65091:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2725.094035] LustreError: 61652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779695710 with bad export cookie 3389497307541195434 [ 2725.097681] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2725.106085] LustreError: 61652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2725.536810] Lustre: server umount lustre-MDT0001 complete [ 2733.999731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2745.314057] Lustre: 60583:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779695714/real 1779695714] req@000000009d989886 x1866145986585984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779695730 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2745.346595] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2746.395299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2750.501117] LustreError: 60580:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005ccd8d16 x1866145986587008/t0(0) o250->MGC192.168.201.157@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2750.915849] LustreError: 66036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2751.031233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2751.105241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2754.094569] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2756.129734] Lustre: 60584:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779695725/real 1779695725] req@00000000f70b79a1 x1866145986586624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779695741 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2756.155520] Lustre: 60584:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2756.587354] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2768.102535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2774.017621] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2774.039830] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2783.607188] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2789.912527] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2799.588690] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2799.592098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2799.619898] Lustre: Skipped 1 previous similar message [ 2804.195537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2806.167805] Lustre: server umount lustre-MDT0000 complete [ 2809.832607] LustreError: 67549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2809.881450] LustreError: 67549:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 2811.590481] LustreError: 61652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779695796 with bad export cookie 3389497307541196813 [ 2811.592125] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2811.601905] LustreError: 61652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2812.034765] Lustre: server umount lustre-MDT0001 complete [ 2819.540767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2823.798773] Lustre: DEBUG MARKER: files compared the same [ 2836.371941] Lustre: server umount lustre-OST0000 complete [ 2860.298544] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 2863.284480] Key type lgssc unregistered [ 2863.651103] LNet: 70587:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2863.667983] LNetError: 70587:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2863.685323] LNet: Removed LNI 192.168.201.157@tcp [ 2865.061902] Key type .llcrypt unregistered [ 2865.063956] Key type ._llcrypt unregistered [ 2898.274651] Key type ._llcrypt registered [ 2898.279847] Key type .llcrypt registered [ 2898.464141] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 2900.029525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2900.213895] alg: No test for adler32 (adler32-zlib) [ 2901.616835] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 2901.863920] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 2903.785045] Key type lgssc registered [ 2905.997787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2917.917705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2917.958586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2919.733538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2926.230606] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug +malloc all [ 2929.128565] LustreError: 71991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2929.182940] LustreError: 71991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2934.245256] LustreError: 71990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2937.331394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2937.652634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2943.869723] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug +malloc all [ 2951.185371] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2956.773862] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2965.660430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2966.100820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2973.342380] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug +malloc all [ 2980.687332] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2989.706096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 3003.363734] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3003.383093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3003.395262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3005.015779] Lustre: server umount lustre-OST0000 complete [ 3010.530558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3010.569424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3010.578822] Lustre: Skipped 1 previous similar message [ 3015.650448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3015.663890] Lustre: Skipped 1 previous similar message [ 3020.768665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3020.773598] Lustre: Skipped 1 previous similar message [ 3024.352368] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3024.590401] Lustre: server umount lustre-MDT0000 complete [ 3025.893663] LustreError: 71990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3025.919617] LustreError: 71990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3030.184593] LustreError: 71971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779696015 with bad export cookie 11038285187515158760 [ 3030.193264] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3030.214169] LustreError: 71971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3030.869790] Lustre: server umount lustre-MDT0001 complete [ 3051.226973] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 3055.926668] Key type lgssc unregistered [ 3056.586233] LNet: 75215:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3056.615648] LNetError: 75215:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3056.644213] LNet: Removed LNI 192.168.201.157@tcp [ 3057.954270] Key type .llcrypt unregistered [ 3057.958087] Key type ._llcrypt unregistered [ 3096.918564] Key type ._llcrypt registered [ 3096.920830] Key type .llcrypt registered [ 3097.066470] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 3098.692513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3098.805602] alg: No test for adler32 (adler32-zlib) [ 3100.134974] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 3100.479696] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 3102.321364] Key type lgssc registered [ 3104.328554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3115.770994] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3115.813498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3131.492277] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3136.674801] LustreError: 76605:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.157@tcp: failed processing log, type 4: rc = -110 [ 3163.488152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3173.744530] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3186.240677] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3200.181405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3200.274152] Lustre: *** cfs_fail_loc=706, val=0*** [ 3200.828532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3200.956081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 3206.127718] LustreError: 76630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3206.168716] LustreError: 76630:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3208.282875] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3211.247571] LustreError: 76629:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3216.355471] LustreError: 76630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3220.687950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3221.328470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3227.362767] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3235.321141] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3241.135565] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3247.073594] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3247.081643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3247.089578] LustreError: Skipped 1 previous similar message [ 3247.096750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3247.114077] Lustre: Skipped 1 previous similar message [ 3247.131162] Lustre: Skipped 1 previous similar message [ 3249.127442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3249.136053] Lustre: Skipped 1 previous similar message [ 3250.718645] Lustre: server umount lustre-OST0000 complete [ 3257.320593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3257.347818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3257.371324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3259.370932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3259.397093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3259.405554] Lustre: Skipped 1 previous similar message [ 3262.958642] Lustre: server umount lustre-MDT0000 complete [ 3264.490125] LustreError: 77584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3264.513328] LustreError: 77584:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3268.410941] LustreError: 78305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779696253 with bad export cookie 7697163410107909989 [ 3268.417963] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3268.422263] LustreError: 78305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3269.140351] Lustre: server umount lustre-MDT0001 complete [ 3286.491994] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 3291.135170] Key type lgssc unregistered [ 3291.453762] LNet: 79869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3291.475057] LNetError: 79869:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3291.515855] LNet: Removed LNI 192.168.201.157@tcp [ 3292.648930] Key type .llcrypt unregistered [ 3292.651267] Key type ._llcrypt unregistered [ 3326.918122] Key type ._llcrypt registered [ 3326.922484] Key type .llcrypt registered [ 3327.052442] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 3329.376680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3329.397358] alg: No test for adler32 (adler32-zlib) [ 3330.797674] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 3331.261893] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 3333.060446] Key type lgssc registered [ 3334.785437] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3347.000392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3347.024426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3354.720145] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3366.510122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3367.240606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3372.517451] LustreError: 81942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3372.552841] LustreError: 81942:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3374.629225] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3377.636088] LustreError: 81945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3377.658916] LustreError: 81945:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3382.762266] LustreError: 81946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3382.813054] LustreError: 81946:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3387.886978] LustreError: 81941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3387.915134] LustreError: 81941:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3389.412693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3389.786915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3392.878415] LustreError: 81940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3392.921943] LustreError: 81940:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3396.976374] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3403.238488] LustreError: 81942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3403.279457] LustreError: 81942:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3410.856132] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3410.894563] Lustre: MGS: Client 6f6378e1-d1a4-451d-bc88-b75da3d043f2 (at 0@lo) reconnecting [ 3410.945515] LustreError: 81956:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3411.509263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3411.709480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 3419.849603] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3442.145858] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3442.152599] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3442.163807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3442.179292] Lustre: Skipped 1 previous similar message [ 3442.200741] Lustre: Skipped 1 previous similar message [ 3446.199533] Lustre: server umount lustre-OST0000 complete [ 3452.386866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3452.390052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3452.397883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3452.450188] Lustre: Skipped 1 previous similar message [ 3457.511269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3457.520378] Lustre: Skipped 2 previous similar messages [ 3458.331180] Lustre: server umount lustre-MDT0000 complete [ 3462.632406] LustreError: 81941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3462.654646] LustreError: 81941:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3465.107474] Lustre: server umount lustre-MDT0001 complete [ 3469.577546] Lustre: server umount MGS complete [ 3488.097626] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 04:07:50 (1779696470) [ 3535.254447] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 3552.013553] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 3610.776559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3618.040548] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3632.221907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3632.795419] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3632.884607] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3633.107795] Lustre: lustre-MDT0001: new disk, initializing [ 3633.234685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3633.286905] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3633.301505] Lustre: 89070:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3633.318739] LustreError: 89070:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3639.914300] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3656.614751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3657.272416] Lustre: lustre-OST0000: new disk, initializing [ 3657.279800] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3657.312041] Lustre: 89823:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3657.516473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3666.604544] Lustre: 90079:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000005843ecf4 x1866146844293248/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1779696667 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3668.406161] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3680.581520] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3687.393792] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3692.673714] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3692.691485] Lustre: MGS: Client 79069c2f-610f-4583-afaa-e2caf677bf1f (at 0@lo) reconnecting [ 3692.749673] Lustre: 90758:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3692.771133] LustreError: 89056:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3692.779668] LustreError: 89056:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3692.802969] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3692.886599] Lustre: lustre-MDT0000: new disk, initializing [ 3693.002289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3693.028060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3694.118735] Lustre: 90773:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000001513ceab x1866146844301184/t0(0) o700->lustre-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1779696695 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3694.179756] Lustre: 90773:0:(client.c:1638:after_reply()) Skipped 1 previous similar message [ 3699.233722] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3700.898641] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3702.770943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3702.789874] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3703.295645] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 3703.375993] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 3733.481480] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3733.496656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3733.516995] Lustre: Skipped 1 previous similar message [ 3733.542155] Lustre: Skipped 1 previous similar message [ 3736.636842] Lustre: server umount lustre-OST0000 complete [ 3743.719518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3743.725619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3743.761225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3749.092746] Lustre: server umount lustre-MDT0000 complete [ 3752.944072] LustreError: 91190:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3752.986820] LustreError: 91190:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3770.336418] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3771.380161] Lustre: server umount lustre-MDT0001 complete [ 3778.322632] Lustre: server umount MGS complete [ 3797.210429] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 04:12:58 (1779696778) [ 3826.774436] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 3831.559209] Key type lgssc unregistered [ 3831.912825] LNet: 93252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3831.933840] LNetError: 93252:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3831.975464] LNet: Removed LNI 192.168.201.157@tcp [ 3832.996298] Key type .llcrypt unregistered [ 3832.999540] Key type ._llcrypt unregistered [ 3860.415580] Key type ._llcrypt registered [ 3860.418410] Key type .llcrypt registered [ 3860.770609] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 3863.555605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3863.646416] alg: No test for adler32 (adler32-zlib) [ 3865.005615] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 3865.590821] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 3867.385191] Key type lgssc registered [ 3869.280636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3882.419412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3882.437428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3884.732630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3891.446302] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3894.247953] LustreError: 94464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3894.287977] LustreError: 94464:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3899.368715] LustreError: 94465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3904.461988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3904.486744] LustreError: 94464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3905.111353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3911.637386] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3919.802752] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3925.287705] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3934.128141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3934.498293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3943.178990] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3952.205092] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3953.994811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3969.518473] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3969.524411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3969.526375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3969.534263] Lustre: Skipped 1 previous similar message [ 3974.088831] Lustre: server umount lustre-OST0000 complete [ 3980.118364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3980.148747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3980.163308] Lustre: Skipped 2 previous similar messages [ 3985.378827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3985.397581] Lustre: Skipped 1 previous similar message [ 3986.577965] Lustre: server umount lustre-MDT0000 complete [ 3989.482980] LustreError: 94465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3989.526427] LustreError: 94465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3992.697036] LustreError: 94446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779696977 with bad export cookie 14456357029451768355 [ 3992.699522] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3992.708216] LustreError: 94446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3993.214951] Lustre: server umount lustre-MDT0001 complete [ 4011.057537] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 4015.718861] Key type lgssc unregistered [ 4016.252885] LNet: 97666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4016.268196] LNetError: 97666:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4016.295963] LNet: Removed LNI 192.168.201.157@tcp [ 4017.383535] Key type .llcrypt unregistered [ 4017.391844] Key type ._llcrypt unregistered [ 4039.309937] Key type ._llcrypt registered [ 4039.316834] Key type .llcrypt registered [ 4039.527261] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 4041.584634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4041.698187] alg: No test for adler32 (adler32-zlib) [ 4043.100593] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 4043.647069] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 4045.448242] Key type lgssc registered [ 4046.723780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4056.288641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4056.302764] LustreError: 98726:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 4056.376902] LustreError: 98733:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 4056.395575] LustreError: 98726:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4916 [ 4056.410752] LustreError: 98726:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 4056.433244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4056.440480] LustreError: 98726:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 4058.497981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4068.840164] LustreError: 98770:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4068.865038] LustreError: 98770:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4069.745318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4070.201941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4077.705050] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4086.624049] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing lsmod [ 4091.931121] LustreError: 99908:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 4091.976781] LustreError: 99909:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 4091.984888] LustreError: 99908:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4956 [ 4091.990425] LustreError: 99908:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 4092.002387] LustreError: 99908:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 4092.022182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4092.536925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4101.093555] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4101.100312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4101.104292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4101.136937] Lustre: Skipped 2 previous similar messages [ 4102.938416] Lustre: server umount lustre-MDT0000 complete [ 4106.214240] LustreError: 98771:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4106.238416] LustreError: 98771:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4108.054850] LustreError: 98752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779697093 with bad export cookie 17385106731261166310 [ 4108.082498] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4108.084746] LustreError: 98752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4108.712824] Lustre: server umount lustre-MDT0001 complete [ 4123.317176] Lustre: server umount lustre-OST0000 complete [ 4137.509204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4138.573142] LustreError: 100973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4138.719416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4144.116115] LustreError: 100974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4147.382471] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4148.199718] LustreError: 100973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4158.436255] LustreError: 100974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4158.467321] LustreError: 100974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4163.159668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4163.560143] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4163.572191] Lustre: Skipped 2 previous similar messages [ 4163.933761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4170.008280] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4179.906144] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4186.214663] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4196.818329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4197.503269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4206.447923] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4208.792397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 4214.602317] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4224.489858] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4224.490363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4224.504049] LustreError: Skipped 1 previous similar message [ 4224.509607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4224.532181] Lustre: Skipped 1 previous similar message [ 4224.590981] Lustre: Skipped 1 previous similar message [ 4229.448580] Lustre: server umount lustre-OST0000 complete [ 4235.754389] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4235.770893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4235.804712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4239.844479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4239.850302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4241.482727] Lustre: server umount lustre-MDT0000 complete [ 4244.003536] LustreError: 100973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4244.051488] LustreError: 100973:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4246.404498] LustreError: 100955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779697231 with bad export cookie 17385106731261167003 [ 4246.426715] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4246.930135] Lustre: server umount lustre-MDT0001 complete [ 4261.918698] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 4264.893327] Key type lgssc unregistered [ 4265.192261] LNet: 104156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4265.201786] LNetError: 104156:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4265.224335] LNet: Removed LNI 192.168.201.157@tcp [ 4266.305413] Key type .llcrypt unregistered [ 4266.308505] Key type ._llcrypt unregistered [ 4297.928554] Key type ._llcrypt registered [ 4297.931333] Key type .llcrypt registered [ 4298.157466] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 4299.773920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4299.802395] alg: No test for adler32 (adler32-zlib) [ 4301.036993] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 4301.442759] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 4303.256961] Key type lgssc registered [ 4304.932531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4314.837865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4314.874020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4316.643959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4323.235152] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4325.862984] LustreError: 105557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4325.887785] LustreError: 105557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4331.020806] LustreError: 105558:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4336.097295] LustreError: 105557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4336.264768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4336.889515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4344.294388] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4354.343149] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4361.236368] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4370.518805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4370.940041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4379.633529] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4386.325733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 4388.060858] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4394.858357] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 4397.435285] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 4406.763736] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4406.784741] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4406.804658] LustreError: Skipped 1 previous similar message [ 4406.839548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4411.616548] Lustre: server umount lustre-OST0000 complete [ 4416.995394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4417.025631] Lustre: Skipped 1 previous similar message [ 4417.038151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4417.042491] Lustre: Skipped 2 previous similar messages [ 4422.116726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4422.126975] Lustre: Skipped 1 previous similar message [ 4423.215696] Lustre: server umount lustre-MDT0000 complete [ 4427.234178] LustreError: 105557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4427.257451] LustreError: 105557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4428.335108] LustreError: 105538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779697413 with bad export cookie 15102472712452412124 [ 4428.337869] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4428.346063] LustreError: 105538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4428.764449] Lustre: server umount lustre-MDT0001 complete [ 4443.188537] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 4446.745628] Key type lgssc unregistered [ 4447.056136] LNet: 108844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4447.069552] LNetError: 108844:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4447.088757] LNet: Removed LNI 192.168.201.157@tcp [ 4448.103214] Key type .llcrypt unregistered [ 4448.106092] Key type ._llcrypt unregistered [ 4472.033415] Key type ._llcrypt registered [ 4472.038565] Key type .llcrypt registered [ 4472.203208] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 4474.326967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4474.581175] alg: No test for adler32 (adler32-zlib) [ 4475.859490] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 4476.233763] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 4478.048387] Key type lgssc registered [ 4480.258157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4491.855162] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4491.929338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4493.615487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4499.782956] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4503.021917] LustreError: 110049:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4503.058539] LustreError: 110049:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4508.145708] LustreError: 110050:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4512.317792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4512.819691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4518.834059] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4527.252331] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4532.556083] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4541.642207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4542.079616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4549.360675] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4556.591939] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4564.606912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 4578.280056] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4578.290132] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4578.315174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4580.324893] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4580.337986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4580.345651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4582.746726] Lustre: server umount lustre-OST0000 complete [ 4589.538288] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4589.547799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4589.570231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4589.577813] Lustre: Skipped 1 previous similar message [ 4595.695109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4595.711113] Lustre: Skipped 3 previous similar messages [ 4600.803821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4600.819193] Lustre: Skipped 1 previous similar message [ 4602.336242] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4602.759504] Lustre: server umount lustre-MDT0000 complete [ 4604.908780] LustreError: 110946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4604.931394] LustreError: 110946:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4607.591984] LustreError: 110032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779697592 with bad export cookie 14147799879181053218 [ 4607.612810] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4608.116860] Lustre: server umount lustre-MDT0001 complete [ 4624.924792] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 4628.955727] Key type lgssc unregistered [ 4629.431147] LNet: 113257:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4629.457684] LNetError: 113257:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4629.499728] LNet: Removed LNI 192.168.201.157@tcp [ 4630.649156] Key type .llcrypt unregistered [ 4630.651418] Key type ._llcrypt unregistered [ 4668.153301] Key type ._llcrypt registered [ 4668.164412] Key type .llcrypt registered [ 4668.357677] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 4669.207679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4669.260349] alg: No test for adler32 (adler32-zlib) [ 4670.387929] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 4670.683928] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 4672.408832] Key type lgssc registered [ 4674.128177] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4684.134815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4684.170238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4685.956048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4691.751912] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4696.548829] LustreError: 114659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4696.614213] LustreError: 114659:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4701.669799] LustreError: 114658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4703.055631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4703.853288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4710.194197] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4718.245989] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4723.436300] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4731.798663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4732.395817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4740.626234] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4748.208481] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4756.011198] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 4766.965762] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4771.275245] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4771.286674] Lustre: Skipped 2 previous similar messages [ 4774.520112] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4774.575202] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4776.225665] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4776.231175] Lustre: Skipped 2 previous similar messages [ 4781.147941] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4781.149596] Lustre: Skipped 1 previous similar message [ 4786.536770] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.57@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4786.545427] Lustre: Skipped 2 previous similar messages [ 4788.825433] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.57@tcp 192.168.0.[2,10]@tcp [ 4793.652710] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.57/24@tcp in log lustre-MDT0000 [ 4793.660733] Lustre: Skipped 2 previous similar messages [ 4796.185574] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.57/24@tcp [ 4796.193062] Lustre: Skipped 2 previous similar messages [ 4807.139070] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4807.146217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4807.176191] Lustre: Skipped 1 previous similar message [ 4807.199892] Lustre: Skipped 1 previous similar message [ 4811.885263] Lustre: server umount lustre-OST0000 complete [ 4816.360194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4816.372531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4816.399974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4817.379508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4817.404535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4817.417640] Lustre: Skipped 1 previous similar message [ 4822.508646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4822.521516] Lustre: Skipped 1 previous similar message [ 4827.621388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4827.636914] Lustre: Skipped 1 previous similar message [ 4830.176201] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4830.427990] Lustre: server umount lustre-MDT0000 complete [ 4832.738147] LustreError: 115550:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4832.766868] LustreError: 115550:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4835.413746] LustreError: 114639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779697820 with bad export cookie 3943293669794503256 [ 4835.416130] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4835.436629] LustreError: 114639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4835.860823] Lustre: server umount lustre-MDT0001 complete [ 4850.831997] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 4855.060968] Key type lgssc unregistered [ 4855.653744] LNet: 119083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4855.670699] LNetError: 119083:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4855.716794] LNet: Removed LNI 192.168.201.157@tcp [ 4857.143386] Key type .llcrypt unregistered [ 4857.145775] Key type ._llcrypt unregistered [ 4893.720362] Key type ._llcrypt registered [ 4893.727208] Key type .llcrypt registered [ 4893.869031] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 4895.267506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4895.367495] alg: No test for adler32 (adler32-zlib) [ 4896.878375] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 4897.364234] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 4899.208177] Key type lgssc registered [ 4900.599208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4911.015872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4911.033297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4912.847502] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4912.895402] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.57/24@tcp [ 4912.911405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4918.759707] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4922.340126] LustreError: 120628:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4922.360035] LustreError: 120628:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4927.466473] LustreError: 120627:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4930.993676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4931.349763] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4931.381054] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.57/24@tcp [ 4931.452221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4938.140142] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4946.825040] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4953.322986] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4962.943658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4963.621937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4972.087881] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4980.947701] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4982.110843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 4996.589609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4996.607176] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4996.639103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4997.601732] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4997.614359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4997.624213] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5002.738555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5002.762807] Lustre: Skipped 2 previous similar messages [ 5007.860554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5007.867232] Lustre: Skipped 1 previous similar message [ 5008.864631] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5009.170786] Lustre: server umount lustre-OST0000 complete [ 5018.091833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5018.118671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5018.120235] Lustre: Skipped 1 previous similar message [ 5018.136596] Lustre: Skipped 1 previous similar message [ 5020.843929] Lustre: server umount lustre-MDT0000 complete [ 5023.201926] LustreError: 120627:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5023.229099] LustreError: 120627:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5026.045618] LustreError: 120608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779698011 with bad export cookie 14138017556816757162 [ 5026.086983] LustreError: 120608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5026.094704] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5026.691328] Lustre: server umount lustre-MDT0001 complete [ 5041.982372] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 5046.188587] Key type lgssc unregistered [ 5046.650426] LNet: 123865:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5046.656081] LNetError: 123865:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5046.677424] LNet: Removed LNI 192.168.201.157@tcp [ 5047.887699] Key type .llcrypt unregistered [ 5047.890789] Key type ._llcrypt unregistered [ 5084.469445] Key type ._llcrypt registered [ 5084.477352] Key type .llcrypt registered [ 5084.607349] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 5105.108863] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 5106.783611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5106.894397] alg: No test for adler32 (adler32-zlib) [ 5108.338888] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 5108.607357] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 5110.369098] Key type lgssc registered [ 5111.986819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5160.502236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5160.529709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5161.872536] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5161.902762] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5162.050275] Lustre: lustre-MDT0000: new disk, initializing [ 5162.155799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5162.180839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5168.205960] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5183.254040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5183.438066] Lustre: 127248:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5183.514451] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5183.524260] Lustre: Skipped 1 previous similar message [ 5183.714737] Lustre: lustre-MDT0001: new disk, initializing [ 5183.852914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5183.907349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5183.926200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5189.577432] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5199.149751] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5204.688632] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5214.041111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5214.286673] Lustre: lustre-OST0000: new disk, initializing [ 5214.290925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5214.298795] Lustre: 128993:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5214.404769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5221.929082] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5224.085481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5224.100691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5224.237317] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5232.652660] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5239.341537] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5239.603640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5245.578889] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5245.835922] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5250.017760] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5250.032295] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5250.046045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5250.065537] Lustre: Skipped 1 previous similar message [ 5254.660412] Lustre: server umount lustre-OST0000 complete [ 5260.259224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5260.264050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5260.273381] Lustre: Skipped 1 previous similar message [ 5260.297329] Lustre: Skipped 2 previous similar messages [ 5265.381704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5265.398759] Lustre: Skipped 1 previous similar message [ 5265.672145] Lustre: server umount lustre-MDT0000 complete [ 5270.434183] LustreError: 127238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779698255 with bad export cookie 18064783296650689337 [ 5270.445753] LustreError: 127238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5270.449204] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5271.074762] Lustre: server umount lustre-MDT0001 complete [ 5282.862804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5283.372268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5288.418750] LustreError: 131062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5288.437982] LustreError: 131062:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5288.887247] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5293.541494] LustreError: 131063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5298.673295] LustreError: 131062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5299.626896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5300.042236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5306.019529] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5314.283513] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5320.951733] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5328.456980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5328.783175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5336.671608] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5344.553701] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5350.535500] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5354.190741] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 5359.585780] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5359.879474] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5370.572950] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5370.886551] Lustre: lustre-OST0001: new disk, initializing [ 5370.894338] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5370.909092] Lustre: 134083:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5371.075824] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5377.746697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5377.758654] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5377.879088] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5380.299817] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5392.797506] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5393.116695] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5398.577285] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5398.833081] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5404.054823] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90c147ced800.ost_server_uuid 50 [ 5406.436629] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90c147ced800.ost_server_uuid in IDLE state after 0 sec [ 5418.986084] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5418.990404] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5419.000247] LustreError: Skipped 2 previous similar messages [ 5419.003715] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5419.018307] Lustre: Skipped 1 previous similar message [ 5419.036538] Lustre: Skipped 1 previous similar message [ 5423.795691] Lustre: server umount lustre-OST0001 complete [ 5424.121317] LustreError: 132660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5424.141920] LustreError: 132660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5429.217548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5429.225595] LustreError: 132660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5429.236335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5429.251944] Lustre: Skipped 1 previous similar message [ 5429.297584] LustreError: 132660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5439.468067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5439.475019] Lustre: Skipped 3 previous similar messages [ 5439.483962] LustreError: 132660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5439.526077] LustreError: 132660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5443.040144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5443.156191] Lustre: server umount lustre-OST0000 complete [ 5448.676669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5448.701475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5454.702508] Lustre: server umount lustre-MDT0000 complete [ 5459.937131] LustreError: 131063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5459.960549] LustreError: 131063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5460.296930] LustreError: 131042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779698445 with bad export cookie 18064783296650690219 [ 5460.307860] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5460.314051] LustreError: 131042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5460.751630] Lustre: server umount lustre-MDT0001 complete [ 5487.254920] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 5490.574505] Key type lgssc unregistered [ 5490.828560] LNet: 136924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5490.836888] LNetError: 136924:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5490.859079] LNet: Removed LNI 192.168.201.157@tcp [ 5491.930446] Key type .llcrypt unregistered [ 5491.933949] Key type ._llcrypt unregistered [ 5555.137731] Key type ._llcrypt registered [ 5555.146229] Key type .llcrypt registered [ 5555.415774] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 5576.982721] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 5578.892765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5578.965213] alg: No test for adler32 (adler32-zlib) [ 5580.226696] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 5580.658581] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 5582.528174] Key type lgssc registered [ 5584.047917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5631.728143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5631.773672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5633.160230] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5633.217604] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5633.381344] Lustre: lustre-MDT0000: new disk, initializing [ 5633.543153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5633.620174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5638.944517] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5654.677800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5654.872985] Lustre: 141053:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5655.030177] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5655.044607] Lustre: Skipped 1 previous similar message [ 5655.166922] Lustre: lustre-MDT0001: new disk, initializing [ 5655.286657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5655.379670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5655.414158] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5661.407871] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5673.020916] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5679.852507] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5690.622390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5691.212708] Lustre: lustre-OST0000: new disk, initializing [ 5691.236087] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5691.270627] Lustre: 142795:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5691.486757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5699.604774] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5700.720493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5700.732439] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5700.944988] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5710.063417] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5715.660820] Lustre: Failing over lustre-OST0000 [ 5715.942689] Lustre: server umount lustre-OST0000 complete [ 5732.824314] Lustre: 139276:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779698701/real 1779698701] req@00000000428a2376 x1866149202528128/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1779698717 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5732.835637] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5732.867227] Lustre: 139276:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5736.992835] Lustre: 139276:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779698706/real 1779698706] req@00000000e8f4bd9b x1866149202530176/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779698722 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5738.118139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5738.790265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5738.803253] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 5741.858366] Lustre: 139276:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779698711/real 1779698711] req@000000007a54c1f0 x1866149202531200/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779698727 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5741.924224] Lustre: 139276:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5742.892532] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5744.088556] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5744.091356] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 5747.234168] Lustre: 139278:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779698716/real 1779698716] req@00000000eae3c155 x1866149202532352/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779698732 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5747.613273] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5752.477630] Lustre: Failing over lustre-MDT0000 [ 5752.771325] Lustre: server umount lustre-MDT0000 complete [ 5752.815353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5752.833890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5752.851043] Lustre: Skipped 1 previous similar message [ 5758.442852] LustreError: 141063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5758.468873] LustreError: 141063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 5759.407025] LustreError: 141058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5763.557083] LustreError: 143172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5763.586103] LustreError: 143172:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5768.686659] LustreError: 141064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5768.713272] LustreError: 141064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5769.702349] Lustre: 139275:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779698738/real 1779698738] req@000000008b6154f9 x1866149202543616/t0(0) o400->MGC192.168.201.157@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1779698754 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5769.738889] Lustre: 139275:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5769.747151] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5774.266410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5774.820566] LustreError: 141064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5774.849792] LustreError: 141064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5778.913488] LustreError: 139274:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007efb5eec x1866149202548864/t0(0) o250->MGC192.168.201.157@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5779.262826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5779.321849] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5779.872860] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5784.579151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5784.600458] Lustre: Skipped 1 previous similar message [ 5784.644797] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5784.713276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5784.748356] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5795.305555] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5795.321886] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5795.323789] LustreError: Skipped 1 previous similar message [ 5795.328895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5795.345620] Lustre: Skipped 3 previous similar messages [ 5795.365924] Lustre: Skipped 1 previous similar message [ 5799.922040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5799.930057] Lustre: Skipped 1 previous similar message [ 5801.486292] Lustre: server umount lustre-OST0000 complete [ 5810.152983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5810.159190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5810.159626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5810.186660] Lustre: Skipped 1 previous similar message [ 5815.267813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5815.279651] Lustre: Skipped 2 previous similar messages [ 5820.384334] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5820.392696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5820.463457] Lustre: Skipped 1 previous similar message [ 5820.843855] Lustre: server umount lustre-MDT0000 complete [ 5825.511555] LustreError: 142149:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5825.555065] LustreError: 142149:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5827.331401] LustreError: 141045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779698812 with bad export cookie 3500347545395624380 [ 5827.332591] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5827.348054] LustreError: 141045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5827.995448] Lustre: server umount lustre-MDT0001 complete [ 5844.251039] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 5848.320502] Key type lgssc unregistered [ 5848.837928] LNet: 146293:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5848.858328] LNetError: 146293:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5848.881187] LNet: Removed LNI 192.168.201.157@tcp [ 5850.041766] Key type .llcrypt unregistered [ 5850.043770] Key type ._llcrypt unregistered [ 5898.581987] Key type ._llcrypt registered [ 5898.587099] Key type .llcrypt registered [ 5898.785960] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 5901.121113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5901.165184] alg: No test for adler32 (adler32-zlib) [ 5902.452532] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 5902.812955] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 5904.568782] Key type lgssc registered [ 5906.193610] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5915.725416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5915.766335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5917.437615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5923.564764] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5926.886270] LustreError: 147798:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5926.904438] LustreError: 147798:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5932.003133] LustreError: 147799:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5933.494246] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5937.121749] LustreError: 147798:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5938.924829] LustreError: 147827:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5938.934347] LustreError: 147827:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5938.942324] LustreError: 147827:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 5939.616755] Lustre: server umount lustre-MDT0000 complete [ 5962.317639] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 5975.423711] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 6028.210902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6028.469899] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6028.502456] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6028.576873] Lustre: lustre-MDT0000: new disk, initializing [ 6028.661159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6028.677540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6034.639835] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6049.060664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6049.177104] Lustre: 150989:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6049.284864] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6049.289704] Lustre: Skipped 1 previous similar message [ 6049.413985] Lustre: lustre-MDT0001: new disk, initializing [ 6049.532473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6049.630421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6049.656903] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6057.031623] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6067.929711] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6074.017563] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6084.409912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6084.752247] Lustre: lustre-OST0000: new disk, initializing [ 6084.756558] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6084.764030] Lustre: 152731:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6084.915228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6092.888645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6092.915268] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6093.037189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6094.752072] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6105.800729] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6110.858455] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 6113.459078] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 04:51:35 (1779699095) [ 6467.040962] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6467.056309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6467.075078] Lustre: Skipped 1 previous similar message [ 6467.102315] Lustre: Skipped 2 previous similar messages [ 6472.170357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6472.184860] Lustre: Skipped 2 previous similar messages [ 6477.285768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6477.298930] Lustre: Skipped 2 previous similar messages [ 6479.329485] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6479.861818] Lustre: server umount lustre-MDT0000 complete [ 6482.403100] LustreError: 150994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6482.429696] LustreError: 150994:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6485.476996] LustreError: 151767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779699470 with bad export cookie 8078637520474392675 [ 6485.486058] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6485.501653] LustreError: 151767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6487.522845] LustreError: 150996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6487.534661] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6487.556983] Lustre: Skipped 1 previous similar message [ 6487.563290] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6491.942357] Lustre: server umount lustre-MDT0001 complete [ 6504.053775] Lustre: server umount lustre-OST0000 complete [ 6517.304644] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 6533.211646] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 6591.395079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6591.747710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6591.803706] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6591.887458] Lustre: lustre-MDT0000: new disk, initializing [ 6592.017935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6592.047986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6599.009294] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6615.081333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6615.310811] Lustre: 157302:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6615.403991] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6615.410381] Lustre: Skipped 1 previous similar message [ 6615.573682] Lustre: lustre-MDT0001: new disk, initializing [ 6615.793604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6615.907574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6615.946223] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6623.261948] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6635.805774] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6643.961775] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6659.835793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6660.291605] Lustre: lustre-OST0000: new disk, initializing [ 6660.301123] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6660.308462] Lustre: 159046:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6660.451356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6665.930279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6665.953094] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6666.277394] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6671.712462] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6683.741137] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6690.893626] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6691.237644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6698.130086] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6698.548719] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6702.066602] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6702.076530] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6702.102513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6702.128662] Lustre: Skipped 1 previous similar message [ 6707.181576] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6707.213826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6712.297758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6712.318329] Lustre: Skipped 3 previous similar messages [ 6716.386146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6716.690867] Lustre: server umount lustre-OST0000 complete [ 6722.490244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6722.544277] Lustre: Skipped 1 previous similar message [ 6729.032855] Lustre: server umount lustre-MDT0000 complete [ 6732.770966] LustreError: 157313:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6732.796240] LustreError: 157313:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6734.700390] LustreError: 157295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779699719 with bad export cookie 8078637520474551512 [ 6734.706217] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6734.715901] LustreError: 157295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6735.096567] Lustre: server umount lustre-MDT0001 complete [ 6748.100098] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 05:02:10 (1779699730) [ 6771.326966] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 6790.460503] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 6844.740383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6845.277205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6845.463612] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6845.732582] Lustre: lustre-MDT0000: new disk, initializing [ 6845.911425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6845.945162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6853.373396] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6868.775317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6869.012281] Lustre: 163670:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6869.481909] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6876.103313] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6886.659466] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6892.953250] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6904.458142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6904.873268] Lustre: lustre-OST0000: new disk, initializing [ 6904.878980] Lustre: Skipped 1 previous similar message [ 6904.888706] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6904.902615] Lustre: Skipped 2 previous similar messages [ 6904.910833] Lustre: 165424:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6910.610543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6910.622142] Lustre: Skipped 1 previous similar message [ 6910.625757] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6910.737669] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6914.066691] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6925.228646] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6939.638491] Lustre: server umount lustre-OST0000 complete [ 6946.275153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6946.301740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6946.323324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6946.328573] Lustre: Skipped 4 previous similar messages [ 6954.224313] Lustre: server umount lustre-MDT0000 complete [ 6957.029728] LustreError: 163680:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6957.031458] Lustre: 147111:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779699926/real 1779699926] req@00000000cc5a8249 x1866149540953984/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779699942 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6957.073331] LustreError: 163680:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6957.161542] Lustre: 147111:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6960.059978] LustreError: 163662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779699945 with bad export cookie 8078637520474552394 [ 6960.065265] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6960.076509] LustreError: 163662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6960.548339] Lustre: server umount lustre-MDT0001 complete [ 6973.709467] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 05:05:55 (1779699955) [ 6998.359823] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 7012.302679] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 7067.568809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7068.190500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7068.271541] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7068.437707] Lustre: lustre-MDT0000: new disk, initializing [ 7068.604794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7068.615463] Lustre: Skipped 2 previous similar messages [ 7068.652141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7075.562661] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7090.827475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7090.959987] Lustre: 169600:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7091.246856] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7097.305451] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7108.476765] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7114.897041] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7126.693479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7127.185335] Lustre: 171356:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7133.809972] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7133.972723] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7136.518923] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7146.659142] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7159.780866] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7159.783843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7159.795056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7159.795064] Lustre: Skipped 3 previous similar messages [ 7159.817860] LustreError: Skipped 1 previous similar message [ 7159.904480] Lustre: Skipped 3 previous similar messages [ 7164.902045] LustreError: 171335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7165.101781] Lustre: server umount lustre-OST0000 complete [ 7173.094432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7177.841985] Lustre: server umount lustre-MDT0000 complete [ 7183.769291] LustreError: 169591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779700168 with bad export cookie 8078637520474553479 [ 7183.770981] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7183.787816] LustreError: 169591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7184.425745] Lustre: server umount lustre-MDT0001 complete [ 7200.070956] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 7204.578491] Key type lgssc unregistered [ 7205.123927] LNet: 173086:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7205.154484] LNetError: 173086:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7205.173317] LNet: Removed LNI 192.168.201.157@tcp [ 7206.262945] Key type .llcrypt unregistered [ 7206.270076] Key type ._llcrypt unregistered [ 7240.324293] Key type ._llcrypt registered [ 7240.330330] Key type .llcrypt registered [ 7240.516552] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 7243.007210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7243.097929] alg: No test for adler32 (adler32-zlib) [ 7244.251990] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 7244.785772] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 7246.576882] Key type lgssc registered [ 7248.772555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7260.528761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7260.600294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7262.424812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7268.043354] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7271.917058] LustreError: 174494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7271.947964] LustreError: 174494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7277.031285] LustreError: 174495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7280.247672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7280.932442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7287.875436] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7296.863835] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7302.653042] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7314.228130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7314.748334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7322.920866] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7332.177509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 7332.660463] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7355.882032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7355.893135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7355.923559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7357.921348] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7357.927331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7357.952917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7357.980693] Lustre: Skipped 1 previous similar message [ 7360.519260] Lustre: server umount lustre-OST0000 complete [ 7368.162111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7368.169475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7368.173610] Lustre: Skipped 1 previous similar message [ 7368.182225] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7368.188243] Lustre: Skipped 1 previous similar message [ 7372.801125] Lustre: server umount lustre-MDT0000 complete [ 7373.281772] LustreError: 176505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7373.315021] LustreError: 176505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7378.242112] LustreError: 174476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779700363 with bad export cookie 8623952591185025779 [ 7378.268152] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7378.268433] LustreError: 174476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7378.940197] Lustre: server umount lustre-MDT0001 complete [ 7395.699976] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 7400.717139] Key type lgssc unregistered [ 7401.269720] LNet: 177700:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7401.292889] LNetError: 177700:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7401.344127] LNet: Removed LNI 192.168.201.157@tcp [ 7402.865169] Key type .llcrypt unregistered [ 7402.868603] Key type ._llcrypt unregistered [ 7442.896471] Key type ._llcrypt registered [ 7442.902754] Key type .llcrypt registered [ 7443.094767] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 7445.711220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7445.745076] alg: No test for adler32 (adler32-zlib) [ 7446.807460] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 7447.054911] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 7448.848448] Key type lgssc registered [ 7450.543350] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7461.555175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7461.576908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7463.168163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7469.587702] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7473.640345] LustreError: 179104:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7473.671693] LustreError: 179104:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7478.763537] LustreError: 179103:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7481.959098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7482.807715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7490.022674] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7499.436736] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7505.730815] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7516.588556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7517.064101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7526.481919] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7533.595107] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7536.443751] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7543.781063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7543.824755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7543.834577] Lustre: Skipped 1 previous similar message [ 7548.389183] Lustre: lustre-OST0000: Not available for connect from 192.168.201.57@tcp (stopping) [ 7550.237453] Lustre: server umount lustre-OST0000 complete [ 7561.447889] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff90c14578f800.ost_server_uuid 50 [ 7564.792964] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 7584.740578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7584.768595] Lustre: Skipped 1 previous similar message [ 7584.785213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7584.800019] Lustre: Skipped 3 previous similar messages [ 7587.063797] Lustre: server umount lustre-MDT0000 complete [ 7589.860818] LustreError: 179816:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7589.877815] LustreError: 179816:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7591.918266] LustreError: 179084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779700577 with bad export cookie 8591091912031629484 [ 7591.918812] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7591.938036] LustreError: 179084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7592.446847] Lustre: server umount lustre-MDT0001 complete [ 7604.834810] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 05:16:26 (1779700586) [ 7614.767617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7615.451820] LustreError: 182594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7615.575045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7620.585845] LustreError: 182595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7621.847144] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7630.823326] LustreError: 182595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7630.842274] LustreError: 182595:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7634.130765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7634.624754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7640.880805] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7648.206673] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7654.565731] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7666.049337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7666.683674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7677.999883] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7686.177699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7687.094404] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7698.897412] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7699.116110] Lustre: lustre-OST0001: new disk, initializing [ 7699.120660] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7699.126369] Lustre: 185099:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7699.262804] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7705.853036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7705.890847] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7706.105659] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7708.894443] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7720.274891] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7725.035124] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7725.050705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7725.068237] Lustre: Skipped 1 previous similar message [ 7725.090090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7730.814698] Lustre: server umount lustre-OST0000 complete [ 7731.617119] LustreError: 184192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7740.868255] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7741.456540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7749.919695] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7750.182624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7767.521601] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7767.527649] LustreError: 184192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7767.530408] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7767.530415] Lustre: Skipped 1 previous similar message [ 7767.537150] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7767.537158] Lustre: Skipped 3 previous similar messages [ 7767.543509] LustreError: Skipped 1 previous similar message [ 7767.678198] LustreError: 184192:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 7769.632536] Lustre: server umount lustre-OST0001 complete [ 7777.765453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7777.771965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7777.801035] Lustre: Skipped 2 previous similar messages [ 7777.801327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7777.844031] Lustre: Skipped 2 previous similar messages [ 7782.747635] Lustre: server umount lustre-MDT0000 complete [ 7788.656090] LustreError: 182574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779700773 with bad export cookie 8591091912031630555 [ 7788.666290] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7788.677732] LustreError: 182574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7789.086703] Lustre: server umount lustre-MDT0001 complete [ 7826.622951] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 05:20:08 (1779700808) [ 7836.951865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7837.413317] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7837.484923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7837.838036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7845.565915] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7860.786645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7861.004356] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7861.159038] Lustre: 188398:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7868.360671] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7877.719672] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7883.503714] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7893.940322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7894.261252] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7900.454298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 7903.237488] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7912.514527] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7923.471359] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7923.534639] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7923.761604] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7923.771323] Lustre: Skipped 2 previous similar messages [ 7932.169768] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 7933.038972] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7941.971679] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7946.721975] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7946.726188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7946.749425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7946.759305] Lustre: Skipped 1 previous similar message [ 7946.793563] Lustre: Skipped 1 previous similar message [ 7952.332864] Lustre: server umount lustre-OST0000 complete [ 7952.869658] LustreError: 190269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7952.929561] LustreError: 190269:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7973.354743] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7973.361422] LustreError: Skipped 1 previous similar message [ 7973.367905] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7978.241597] Lustre: server umount lustre-OST0001 complete [ 7984.613161] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7984.634887] LustreError: Skipped 1 previous similar message [ 7984.655977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7984.663867] Lustre: Skipped 5 previous similar messages [ 7990.149138] Lustre: server umount lustre-MDT0000 complete [ 7995.666197] LustreError: 189136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779700980 with bad export cookie 8591091912031631962 [ 7995.680624] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7995.685430] LustreError: 189136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7996.099308] Lustre: server umount lustre-MDT0001 complete [ 8035.081303] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 05:23:36 (1779701016) [ 8060.471996] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 8076.887351] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 8135.640375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8136.145827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8136.206369] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8136.330770] Lustre: lustre-MDT0000: new disk, initializing [ 8136.547640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8136.594416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8144.292220] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8161.034547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8161.128158] Lustre: 196040:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8161.207723] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8161.216855] Lustre: Skipped 1 previous similar message [ 8161.315166] Lustre: lustre-MDT0001: new disk, initializing [ 8161.588195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8161.597417] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8168.599482] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8180.316591] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8187.218230] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8197.777052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8198.197328] Lustre: lustre-OST0000: new disk, initializing [ 8198.208462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8198.228936] Lustre: 197782:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8207.395849] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8207.999673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8208.008359] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8208.268907] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8218.002935] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8224.731702] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8225.049727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8232.341761] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8232.730834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8239.077248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8239.086946] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8239.087977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8239.087983] Lustre: Skipped 2 previous similar messages [ 8239.125157] Lustre: Skipped 4 previous similar messages [ 8250.336353] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8250.666629] Lustre: server umount lustre-OST0000 complete [ 8262.896766] Lustre: server umount lustre-MDT0000 complete [ 8264.684567] LustreError: 197279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8264.704733] LustreError: 197279:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 8268.726345] LustreError: 196032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779701253 with bad export cookie 8591091912031633747 [ 8268.739900] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8268.741996] LustreError: 196032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8269.315504] Lustre: server umount lustre-MDT0001 complete [ 8281.094511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8288.262495] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8298.868129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8305.519609] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8313.808969] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8319.706775] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8329.274868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8339.021771] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8347.213347] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8354.142674] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8354.452267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8360.590037] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8360.872671] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8365.031425] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8365.054835] LustreError: Skipped 1 previous similar message [ 8365.077153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8365.105785] Lustre: Skipped 2 previous similar messages [ 8369.778983] Lustre: server umount lustre-OST0000 complete [ 8379.193813] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8386.386820] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8418.360587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8418.860702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8418.871728] Lustre: Skipped 5 previous similar messages [ 8427.084498] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8432.166048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8432.183231] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8434.224871] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8436.171698] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8442.358344] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8442.368378] LustreError: Skipped 1 previous similar message [ 8442.392855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8442.405912] Lustre: Skipped 10 previous similar messages [ 8447.840921] Lustre: server umount lustre-OST0000 complete [ 8461.451782] Lustre: server umount lustre-MDT0000 complete [ 8467.113356] LustreError: 199840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779701452 with bad export cookie 8591091912031634629 [ 8467.115297] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8467.125380] LustreError: 199840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8467.693118] Lustre: server umount lustre-MDT0001 complete [ 8480.331483] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 05:31:02 (1779701462) [ 8489.686808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8496.333473] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8508.183246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8515.325683] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8524.158220] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8529.952556] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8539.862600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8549.642581] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8557.740710] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8564.272244] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8564.641166] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8571.383684] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8571.763875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8581.944696] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8582.301941] Lustre: lustre-OST0001: new disk, initializing [ 8582.305714] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8582.328769] Lustre: 208314:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8589.036728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8589.056331] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8589.393749] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8594.048311] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8606.664782] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8614.162308] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 8614.581752] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8622.128450] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 8622.686345] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8630.246729] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8630.247588] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8630.272370] LustreError: Skipped 1 previous similar message [ 8630.292944] Lustre: Skipped 5 previous similar messages [ 8631.752400] Lustre: server umount lustre-OST0001 complete [ 8635.363527] LustreError: 206909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8635.413712] LustreError: 206909:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 8640.868271] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 8641.130213] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 8648.122442] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 8648.427534] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 8682.521507] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8688.128741] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8688.156517] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8688.159357] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8688.188539] Lustre: Skipped 1 previous similar message [ 8693.846545] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8702.536781] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8705.995059] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8713.727636] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8713.744015] Lustre: Skipped 7 previous similar messages [ 8715.671638] Lustre: server umount lustre-OST0001 complete [ 8743.954717] Lustre: server umount lustre-OST0000 complete [ 8763.360166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8763.659841] Lustre: server umount lustre-MDT0000 complete [ 8769.218097] LustreError: 205292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779701754 with bad export cookie 8591091912031635588 [ 8769.222373] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8769.231839] LustreError: 205292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8775.740820] Lustre: server umount lustre-MDT0001 complete [ 8809.601655] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 05:36:31 (1779701791) [ 8819.076602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8819.385238] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8819.415145] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8826.635943] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8837.209793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8837.349536] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8837.383418] Lustre: 213272:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8844.125205] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8852.316814] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8857.725928] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8866.637495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8866.956616] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8876.064720] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8884.084496] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8896.254470] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8896.410904] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8904.237771] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8912.140635] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8914.577168] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8920.285369] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 8920.583363] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8926.391860] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 8926.625420] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8931.919352] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90c145819000.ost_server_uuid 50 [ 8934.225408] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90c145819000.ost_server_uuid in IDLE state after 0 sec [ 8935.574320] Lustre: Permanently deactivating lustre-OST0001 [ 8935.590617] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 8940.567498] Lustre: Permanently reactivating lustre-OST0001 [ 8940.614068] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 8940.630787] Lustre: Skipped 2 previous similar messages [ 8950.245740] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8950.266274] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8950.268317] LustreError: Skipped 6 previous similar messages [ 8950.292634] Lustre: Skipped 7 previous similar messages [ 8953.519880] Lustre: server umount lustre-OST0001 complete [ 8964.321462] Lustre: server umount lustre-OST0000 complete [ 8975.671591] Lustre: server umount lustre-MDT0000 complete [ 8980.506162] LustreError: 215806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779701965 with bad export cookie 8591091912031636876 [ 8980.513668] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8980.521276] LustreError: 215806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8981.257912] Lustre: server umount lustre-MDT0001 complete [ 9017.170804] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 05:39:58 (1779701998) [ 9029.416461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9029.884918] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9029.973914] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9029.987600] Lustre: Skipped 2 previous similar messages [ 9030.242256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9030.259912] Lustre: Skipped 9 previous similar messages [ 9036.366647] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9047.659230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9047.743033] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9047.767433] Lustre: 219393:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9054.099623] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9062.751328] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9069.565770] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9080.440759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9080.962365] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9081.008156] Lustre: Permanently deactivating lustre-OST0000 [ 9081.034712] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 9087.106892] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9090.913315] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9100.093119] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9110.071644] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9110.148842] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9117.084827] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 9120.497882] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9129.346865] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9132.288951] Lustre: Permanently reactivating lustre-OST0000 [ 9132.311319] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 9132.317467] Lustre: Skipped 2 previous similar messages [ 9138.326649] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 9138.368123] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 51 seconds. I think it's dead, and I am evicting it. exp 00000000d68f6011, cur 1779702123 deadline 1779702102 last 1779702072 [ 9138.493580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9156.686249] Lustre: server umount lustre-OST0001 complete [ 9158.129592] LustreError: 221017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9158.159993] LustreError: 221017:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 46 previous similar messages [ 9168.875499] Lustre: server umount lustre-OST0000 complete [ 9189.345041] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9189.838829] Lustre: server umount lustre-MDT0000 complete [ 9197.933543] LustreError: 219386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779702183 with bad export cookie 8591091912031638549 [ 9197.942966] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9197.961501] LustreError: 219386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9199.136215] Lustre: server umount lustre-MDT0001 complete [ 9220.744491] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 9225.512275] Key type lgssc unregistered [ 9225.973546] LNet: 223827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9225.998508] LNetError: 223827:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9226.026203] LNet: Removed LNI 192.168.201.157@tcp [ 9227.385233] Key type .llcrypt unregistered [ 9227.389501] Key type ._llcrypt unregistered [ 9261.095965] Key type ._llcrypt registered [ 9261.098046] Key type .llcrypt registered [ 9261.269821] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 9262.421774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9262.447659] alg: No test for adler32 (adler32-zlib) [ 9263.603645] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 9263.895479] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 9265.624225] Key type lgssc registered [ 9267.673258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9283.604521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9283.660358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9285.381676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9291.414617] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9295.848596] LustreError: 225281:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9295.865894] LustreError: 225281:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9300.970265] LustreError: 225280:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9303.663335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9303.851643] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9303.929655] Lustre: 225270:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9303.947584] Lustre: Permanently deactivating lustre-MDT0001 [ 9303.975336] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 9304.531130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9311.473781] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9312.855413] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 9319.287214] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9324.648897] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9334.868711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9335.384786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9345.346326] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9349.687299] LustreError: 226895:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9349.722189] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 9354.734255] LustreError: 227362:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9354.796956] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9355.801389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 9360.873445] LustreError: 227362:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9360.940102] LustreError: 227362:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9367.504192] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9368.260656] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9373.705149] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 9373.707169] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 9377.808272] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9387.223132] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9391.158848] Lustre: Permanently reactivating lustre-MDT0001 [ 9391.184393] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 9391.192466] Lustre: Skipped 2 previous similar messages [ 9397.230501] Lustre: lustre-MDT0001: Client 1e5c9e8d-0ef0-458d-89d2-8728e99b5809 (at 192.168.201.57@tcp) reconnecting [ 9397.881524] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9397.897552] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 9397.907038] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 9397.916349] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 9403.744878] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9406.608795] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9414.640920] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 9415.037664] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 9422.754830] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 9423.363703] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 9428.592148] Lustre: Permanently deactivating lustre-MDT0001 [ 9428.638414] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 9428.655507] Lustre: Skipped 2 previous similar messages [ 9437.916456] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 9450.481974] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9450.484938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9450.513333] Lustre: Skipped 3 previous similar messages [ 9450.535203] Lustre: Skipped 3 previous similar messages [ 9452.384293] Lustre: server umount lustre-MDT0000 complete [ 9455.593184] LustreError: 225274:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9455.617898] LustreError: 225274:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9459.335905] LustreError: 225261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779702444 with bad export cookie 5112906558578770779 [ 9459.348951] LustreError: 225261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9459.352731] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9459.948273] Lustre: server umount lustre-MDT0001 complete [ 9466.984337] Lustre: server umount lustre-OST0000 complete [ 9484.953589] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 05:47:46 (1779702466) [ 9501.746421] Lustre: server umount lustre-OST0001 complete [ 9514.461463] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 9529.356870] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 9592.686472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9593.154766] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9593.165401] Lustre: Skipped 2 previous similar messages [ 9593.196618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9593.361646] Lustre: lustre-MDT0000: new disk, initializing [ 9593.642531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9593.686618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9600.428725] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9616.704794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9616.960321] Lustre: 232939:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9617.062448] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9617.066788] Lustre: Skipped 1 previous similar message [ 9617.164278] Lustre: lustre-MDT0001: new disk, initializing [ 9617.369800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9617.494199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9617.515385] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9623.400082] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9634.018255] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9639.608924] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9649.215434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9649.654892] Lustre: lustre-OST0000: new disk, initializing [ 9649.663527] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9649.685958] Lustre: 234685:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9649.906752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9656.439625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9656.490932] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9656.678550] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9660.792690] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9673.752955] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9678.254313] LustreError: 232946:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 9688.342626] LustreError: 232946:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 9688.402812] LustreError: 234139:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 9693.441912] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9693.843919] Lustre: lustre-OST0001: new disk, initializing [ 9693.856732] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9693.867898] Lustre: 235687:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9694.109491] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9698.432333] LustreError: 234139:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 9702.079574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9702.112960] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9702.280491] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9703.356217] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9715.364592] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9717.588726] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9722.852258] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9722.881873] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9722.905836] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9722.911530] Lustre: Skipped 1 previous similar message [ 9722.938071] Lustre: Skipped 1 previous similar message [ 9727.008508] Lustre: server umount lustre-OST0001 complete [ 9727.976150] LustreError: 235973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9728.037556] LustreError: 235973:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9748.460574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9748.467834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9748.487518] Lustre: Skipped 1 previous similar message [ 9748.515070] Lustre: Skipped 1 previous similar message [ 9753.574752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9753.611107] Lustre: Skipped 1 previous similar message [ 9758.696024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9758.713752] Lustre: Skipped 1 previous similar message [ 9762.272174] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9762.629781] Lustre: server umount lustre-OST0000 complete [ 9770.981795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9771.006375] LustreError: Skipped 1 previous similar message [ 9771.021212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9771.034242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9777.746742] Lustre: server umount lustre-MDT0000 complete [ 9779.172977] LustreError: 232951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9779.220046] LustreError: 232951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 9784.020736] LustreError: 232932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779702769 with bad export cookie 5112906558578773593 [ 9784.028028] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9784.042249] LustreError: 232932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9791.189826] Lustre: server umount lustre-MDT0001 complete [ 9815.638938] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 9820.766819] Key type lgssc unregistered [ 9821.253175] LNet: 237617:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9821.267080] LNetError: 237617:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9821.290392] LNet: Removed LNI 192.168.201.157@tcp [ 9822.930510] Key type .llcrypt unregistered [ 9822.934755] Key type ._llcrypt unregistered [ 9884.972636] Key type ._llcrypt registered [ 9884.979112] Key type .llcrypt registered [ 9885.199058] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 9887.384442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9887.487874] alg: No test for adler32 (adler32-zlib) [ 9888.704672] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 9889.052246] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 9890.856910] Key type lgssc registered [ 9892.885870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9905.734686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9905.758505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9907.154824] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9907.209949] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9907.576902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9913.543937] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9926.991334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9927.134195] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9927.200303] Lustre: 239813:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9927.875717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9935.414836] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9943.129824] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9948.554144] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9960.486312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9960.657842] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9961.025401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9966.835288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 9971.506767] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9980.016176] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10046.874920] Lustre: server umount lustre-OST0000 complete [10053.635142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [10064.353728] Lustre: 239137:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779703033/real 1779703033] req@0000000058ba3160 x1866153720859008/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1779703049 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10064.402416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10065.378553] Lustre: 239138:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779703034/real 1779703034] req@00000000ae4b7e85 x1866153720859136/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1779703050 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10065.403967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10070.580801] Lustre: 239135:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779703039/real 1779703039] req@00000000eba20a6d x1866153720860928/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779703055 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10074.870058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10075.669036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10076.662390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [10076.697954] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [10077.691661] Lustre: 239136:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779703044/real 1779703044] req@00000000125b0ea2 x1866153720861952/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779703060 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10077.735347] Lustre: 239136:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10086.006630] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10089.018648] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [10089.069390] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [10095.694378] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10143.201622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10143.217105] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10143.258418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10143.634530] Lustre: server umount lustre-OST0000 complete [10150.388489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10150.398308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10150.401869] Lustre: Skipped 1 previous similar message [10150.418414] Lustre: Skipped 1 previous similar message [10155.493534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10155.509421] Lustre: Skipped 1 previous similar message [10160.611054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10160.630156] Lustre: Skipped 1 previous similar message [10161.104138] Lustre: 239138:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779703130/real 1779703130] req@0000000007e52160 x1866153720894720/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779703146 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10161.146425] Lustre: 239138:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10161.169478] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10164.192227] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10166.487096] Lustre: server umount lustre-MDT0000 complete [10171.360680] Lustre: 239135:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779703140/real 1779703140] req@00000000865f94fc x1866153720897408/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779703156 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10171.364980] LustreError: 239824:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10171.448466] Lustre: 239135:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [10171.577238] LustreError: 239824:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10173.102071] LustreError: 239806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779703158 with bad export cookie 6128127166065254873 [10173.124652] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10173.128629] LustreError: 239806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10173.683524] Lustre: server umount lustre-MDT0001 complete [10197.160357] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [10203.057824] Key type lgssc unregistered [10203.443888] LNet: 244800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10203.448894] LNetError: 244800:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10203.479233] LNet: Removed LNI 192.168.201.157@tcp [10205.330734] Key type .llcrypt unregistered [10205.333787] Key type ._llcrypt unregistered [10242.141282] Key type ._llcrypt registered [10242.143719] Key type .llcrypt registered [10242.325313] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [10243.978575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10244.259197] alg: No test for adler32 (adler32-zlib) [10245.540441] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [10245.920543] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [10247.736906] Key type lgssc registered [10249.713457] Lustre: Echo OBD driver; http://www.lustre.org/ [10262.588645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10262.641203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10264.281723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10271.219817] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10274.794104] LustreError: 246205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10274.814693] LustreError: 246205:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10279.906856] LustreError: 246204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10285.032907] LustreError: 246205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10287.213398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10287.869384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10295.833465] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10304.842035] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10311.232345] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10322.629913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10323.172128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10332.041979] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10334.481221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [10341.747164] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10378.723957] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10378.744323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10378.756034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10380.770409] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10380.774071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10380.802977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10380.855322] Lustre: Skipped 1 previous similar message [10382.359348] Lustre: server umount lustre-OST0000 complete [10389.990276] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10390.004332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10390.035535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10394.776283] Lustre: server umount lustre-MDT0000 complete [10396.142414] LustreError: 246204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10396.189957] LustreError: 246204:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10401.251277] LustreError: 246929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10401.278751] LustreError: 246929:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10401.326843] LustreError: 246186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779703386 with bad export cookie 4373255972632181473 [10401.341868] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10401.346917] LustreError: 246186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10401.964339] Lustre: server umount lustre-MDT0001 complete [10416.708931] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [10420.603534] Key type lgssc unregistered [10421.234994] LNet: 250173:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10421.243532] LNetError: 250173:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10421.263362] LNet: Removed LNI 192.168.201.157@tcp [10422.156311] Key type .llcrypt unregistered [10422.159576] Key type ._llcrypt unregistered [10446.892120] Key type ._llcrypt registered [10446.901922] Key type .llcrypt registered [10447.116548] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [10449.272160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10449.320764] alg: No test for adler32 (adler32-zlib) [10450.522954] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [10450.969448] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [10452.744184] Key type lgssc registered [10454.664309] Lustre: Echo OBD driver; http://www.lustre.org/ [10465.422676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10465.454705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10467.001198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10473.234773] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10477.556468] LustreError: 251379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10477.598891] LustreError: 251379:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10482.661211] LustreError: 251380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10485.157890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10485.814047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10491.561345] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10500.466981] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10506.563035] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10516.251033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10516.761251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10525.893439] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10535.925587] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10537.365775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10555.873565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10555.882223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10555.901032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10557.928320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10557.937967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10557.959994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10559.493578] Lustre: server umount lustre-OST0000 complete [10567.649669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10567.659079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10567.676152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10567.683608] Lustre: Skipped 1 previous similar message [10571.608306] Lustre: server umount lustre-MDT0000 complete [10573.306092] LustreError: 251376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10573.325568] LustreError: 251376:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10577.021623] LustreError: 251360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779703562 with bad export cookie 4067960335417387087 [10577.033260] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10577.037117] LustreError: 251360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10577.535239] Lustre: server umount lustre-MDT0001 complete [10593.549580] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [10597.442908] Key type lgssc unregistered [10597.972596] LNet: 254704:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10597.995312] LNetError: 254704:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10598.013556] LNet: Removed LNI 192.168.201.157@tcp [10599.226127] Key type .llcrypt unregistered [10599.228842] Key type ._llcrypt unregistered [10638.390235] Key type ._llcrypt registered [10638.392622] Key type .llcrypt registered [10638.613030] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [10641.017467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10641.033900] alg: No test for adler32 (adler32-zlib) [10642.442673] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [10642.824129] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [10644.504182] Key type lgssc registered [10646.000790] Lustre: Echo OBD driver; http://www.lustre.org/ [10656.996232] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10657.011522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10658.802819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10665.587837] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10668.002967] LustreError: 256108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10668.039558] LustreError: 256108:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10673.146662] LustreError: 256107:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10678.174171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10678.241649] LustreError: 256108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10678.640556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10684.696835] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10693.625194] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10700.225541] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10709.548244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10710.115859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10718.821321] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10727.582843] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10728.643908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [10764.784034] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10764.790212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10764.804404] Lustre: Skipped 1 previous similar message [10764.807194] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10764.819679] Lustre: Skipped 1 previous similar message [10767.506872] Lustre: server umount lustre-OST0000 complete [10775.014069] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10775.040229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10775.043397] Lustre: Skipped 1 previous similar message [10775.067308] Lustre: Skipped 1 previous similar message [10780.129513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10780.137820] Lustre: Skipped 1 previous similar message [10785.267113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10785.279709] Lustre: Skipped 1 previous similar message [10788.320562] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10788.568429] Lustre: server umount lustre-MDT0000 complete [10790.370530] LustreError: 256102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10790.388235] LustreError: 256102:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10793.819381] LustreError: 256090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779703778 with bad export cookie 8688149229615570020 [10793.823903] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10793.833305] LustreError: 256090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10794.414488] Lustre: server umount lustre-MDT0001 complete [10811.294726] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [10815.174090] Key type lgssc unregistered [10815.641521] LNet: 260084:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10815.651132] LNetError: 260084:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10815.673604] LNet: Removed LNI 192.168.201.157@tcp [10817.089443] Key type .llcrypt unregistered [10817.095664] Key type ._llcrypt unregistered [10841.431447] Key type ._llcrypt registered [10841.434188] Key type .llcrypt registered [10841.581769] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [10843.327875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10843.458894] alg: No test for adler32 (adler32-zlib) [10844.705755] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [10845.085646] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [10846.832666] Key type lgssc registered [10848.327875] Lustre: Echo OBD driver; http://www.lustre.org/ [10859.230985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10859.253102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10860.922060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10868.828052] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10871.275373] LustreError: 261310:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10871.331291] LustreError: 261310:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10876.399938] LustreError: 261312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10881.507150] LustreError: 261314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10883.773457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10884.220331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10892.485909] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10904.834762] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10911.421475] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10921.298740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10921.998269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10929.968533] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10931.360059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [10938.194421] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10955.748059] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10955.761200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10955.787609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10957.285089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10957.287766] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10957.309593] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10959.735047] Lustre: server umount lustre-OST0000 complete [10965.987092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10966.001990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10966.017730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10966.026123] Lustre: Skipped 1 previous similar message [10972.644201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10972.651088] Lustre: Skipped 3 previous similar messages [10977.771662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10977.785368] Lustre: Skipped 1 previous similar message [10979.296231] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10979.606637] Lustre: server umount lustre-MDT0000 complete [10981.858430] LustreError: 261302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10981.902376] LustreError: 261302:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10985.223471] LustreError: 261276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779703970 with bad export cookie 16520653608525879617 [10985.233624] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10985.245742] LustreError: 261276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10986.076436] Lustre: server umount lustre-MDT0001 complete [11004.051039] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [11007.710885] Key type lgssc unregistered [11008.156223] LNet: 264661:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11008.168913] LNetError: 264661:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11008.194032] LNet: Removed LNI 192.168.201.157@tcp [11009.600957] Key type .llcrypt unregistered [11009.603043] Key type ._llcrypt unregistered [11026.159729] Key type ._llcrypt registered [11026.166806] Key type .llcrypt registered [11026.495634] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [11052.579669] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [11073.113472] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [11075.784977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11075.939866] alg: No test for adler32 (adler32-zlib) [11077.429692] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [11077.740717] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [11079.488387] Key type lgssc registered [11082.131082] Lustre: Echo OBD driver; http://www.lustre.org/ [11139.506372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11139.560144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11141.217628] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11141.283526] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11141.414838] Lustre: lustre-MDT0000: new disk, initializing [11141.575509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11141.629698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11148.888741] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11167.065187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11167.373631] Lustre: 268175:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11167.450577] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11167.458531] Lustre: Skipped 1 previous similar message [11167.615494] Lustre: lustre-MDT0001: new disk, initializing [11167.804300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11167.855209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11167.877706] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11174.868426] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11187.200578] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11193.461878] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11202.631376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11203.061985] Lustre: lustre-OST0000: new disk, initializing [11203.065708] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11203.074377] Lustre: 269919:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11203.183095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11212.380245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11212.415666] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11212.561327] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11212.758068] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11223.740835] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11231.189348] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11231.490582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11237.602943] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11237.889369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11243.534543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11243.554262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11243.580411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11243.604514] Lustre: Skipped 1 previous similar message [11248.621480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11248.641404] Lustre: Skipped 1 previous similar message [11253.729647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11253.768230] Lustre: Skipped 1 previous similar message [11255.264441] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11255.787996] Lustre: server umount lustre-OST0000 complete [11263.973437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11263.999428] Lustre: Skipped 1 previous similar message [11264.012728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11264.026506] Lustre: Skipped 1 previous similar message [11269.096101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11269.119033] Lustre: Skipped 1 previous similar message [11269.533342] Lustre: server umount lustre-MDT0000 complete [11274.233400] LustreError: 268165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779704259 with bad export cookie 16239917700303826074 [11274.236336] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11274.241908] LustreError: 268165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11274.993360] Lustre: server umount lustre-MDT0001 complete [11287.876632] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 06:17:50 (1779704270) [11299.235420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11300.284573] LustreError: 272189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11300.318667] LustreError: 272189:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11300.463659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11305.455855] LustreError: 272190:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11307.645649] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11310.568386] LustreError: 272189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11315.687075] LustreError: 272190:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11320.816604] LustreError: 272189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11322.319241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11322.936579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11330.435590] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11339.054252] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11344.080637] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11355.176211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11355.634177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11364.241948] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11372.923684] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11390.450228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11390.454071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11390.454795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11390.454800] LustreError: Skipped 1 previous similar message [11390.473590] Lustre: Skipped 2 previous similar messages [11390.513381] Lustre: Skipped 1 previous similar message [11392.249476] Lustre: server umount lustre-OST0000 complete [11399.648911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11399.660101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11404.417676] Lustre: server umount lustre-MDT0000 complete [11405.799462] LustreError: 272189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11405.827111] LustreError: 272189:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11410.388679] LustreError: 272908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779704395 with bad export cookie 16239917700303826956 [11410.398253] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11410.409268] LustreError: 272908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11410.930272] Lustre: server umount lustre-MDT0001 complete [11430.878893] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [11434.635592] Key type lgssc unregistered [11434.995422] LNet: 275414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11435.003864] LNetError: 275414:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11436.071533] LNet: Removed LNI 192.168.201.157@tcp [11437.481906] Key type .llcrypt unregistered [11437.484684] Key type ._llcrypt unregistered [11482.317607] Key type ._llcrypt registered [11482.319252] Key type .llcrypt registered [11482.510663] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [11504.790601] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [11506.853191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11507.096408] alg: No test for adler32 (adler32-zlib) [11508.420901] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [11508.688950] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [11510.395154] Key type lgssc registered [11512.027231] Lustre: Echo OBD driver; http://www.lustre.org/ [11582.239342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11582.311950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11583.878735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11583.967269] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11584.160333] Lustre: lustre-MDT0000: new disk, initializing [11584.347616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11584.391576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11592.288539] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11608.519766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11608.710964] Lustre: 279214:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11608.764832] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11608.768450] Lustre: Skipped 1 previous similar message [11608.825942] Lustre: lustre-MDT0001: new disk, initializing [11609.005465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11609.128693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11609.164792] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11617.997848] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11631.582486] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11637.779492] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11648.869375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11649.247712] Lustre: lustre-OST2710: new disk, initializing [11649.260709] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [11649.274558] Lustre: 280961:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11649.471320] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [11654.856597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [11654.870437] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [11655.086116] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [11658.989300] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11670.278654] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [11680.333932] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11680.566485] Lustre: lustre-OST03e8: new disk, initializing [11680.576518] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [11680.585981] Lustre: 281913:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11680.742450] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [11690.075972] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11690.147083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [11690.177331] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [11690.430247] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [11697.341552] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [11707.512783] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [11707.957895] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11715.072832] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [11715.579760] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11721.975706] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [11722.366586] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11729.762074] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [11730.117529] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11741.689547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11741.711094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11741.730310] Lustre: Skipped 2 previous similar messages [11741.749940] Lustre: Skipped 1 previous similar message [11743.055895] Lustre: server umount lustre-MDT0000 complete [11750.048615] LustreError: 279207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779704735 with bad export cookie 1214976383842922770 [11750.060035] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11750.083080] LustreError: 279207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11750.916049] Lustre: server umount lustre-MDT0001 complete [11768.097345] Lustre: 277011:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779704737/real 1779704737] req@00000000b615a7e2 x1866155418530944/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779704753 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11768.183356] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11768.225688] Lustre: Skipped 1 previous similar message [11768.379286] Lustre: server umount lustre-OST2710 complete [11772.142318] Lustre: 277013:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779704741/real 1779704741] req@00000000beca4ddd x1866155418531200/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779704757 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11773.416241] Lustre: 277012:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779704742/real 1779704742] req@0000000069c8e82c x1866155418531456/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779704758 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11775.460470] Lustre: server umount lustre-OST03e8 complete [11805.525794] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [11824.416124] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [11887.978372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11888.347261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11888.451896] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11888.623573] Lustre: lustre-MDT0000: new disk, initializing [11888.852400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11888.928329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11896.647660] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11913.656715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11913.926988] Lustre: 286981:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11914.079938] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11914.096113] Lustre: Skipped 1 previous similar message [11914.289207] Lustre: lustre-MDT0001: new disk, initializing [11914.473332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11914.590146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11914.616502] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11921.397572] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11933.520707] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11939.847290] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11949.651569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11949.933497] Lustre: lustre-OST0000: new disk, initializing [11949.939761] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11949.947648] Lustre: 288725:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11950.123796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11955.903040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11955.925690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11956.102437] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11959.676467] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11972.166539] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11979.904652] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11980.182309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11986.865982] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11987.323463] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11992.034487] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11992.036072] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11992.052763] LustreError: Skipped 1 previous similar message [11992.058562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11992.071295] Lustre: Skipped 1 previous similar message [11992.110144] Lustre: Skipped 3 previous similar messages [11996.975621] Lustre: server umount lustre-OST0000 complete [12002.162024] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12002.181108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12002.196758] Lustre: Skipped 1 previous similar message [12007.394747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12007.413406] Lustre: Skipped 1 previous similar message [12008.863949] Lustre: server umount lustre-MDT0000 complete [12012.524055] LustreError: 288319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12012.551123] LustreError: 288319:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [12014.023398] LustreError: 287759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779704999 with bad export cookie 1214976383842924464 [12014.039533] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12014.057995] LustreError: 287759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12014.549243] Lustre: server umount lustre-MDT0001 complete [12028.453304] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 06:30:10 (1779705010) [12031.120606] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [12033.867875] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 06:30:15 (1779705015) [12038.795582] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [12081.697629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12082.162147] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12082.266481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12082.770502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12089.961170] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12101.826870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12101.910357] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12101.958500] Lustre: 292425:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12108.212657] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12132.665793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12132.895659] Lustre: Denying initial registration attempt from nid 192.168.201.157@tcp, specified as failover [12132.923789] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [12132.936254] LustreError: 293845:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [12132.950494] LustreError: 293845:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [12132.958626] LustreError: 293845:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [12133.026057] Lustre: server umount lustre-OST0000 complete [12133.029130] LustreError: 293845:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [12143.604644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12143.611078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12143.631567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12143.666897] Lustre: Skipped 2 previous similar messages [12151.178941] Lustre: server umount lustre-MDT0000 complete [12153.825392] LustreError: 292431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12153.852593] LustreError: 292431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12157.097498] LustreError: 293150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779705142 with bad export cookie 1214976383842925346 [12157.106994] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12157.109648] LustreError: 293150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12157.780276] Lustre: server umount lustre-MDT0001 complete [12174.481462] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [12179.512982] Key type lgssc unregistered [12179.996494] LNet: 294953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12180.011446] LNetError: 294953:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12180.027905] LNet: Removed LNI 192.168.201.157@tcp [12181.181298] Key type .llcrypt unregistered [12181.188542] Key type ._llcrypt unregistered [12197.753234] Key type ._llcrypt registered [12197.761689] Key type .llcrypt registered [12197.947912] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [12200.275864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12200.375404] alg: No test for adler32 (adler32-zlib) [12201.771205] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [12202.210086] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [12203.944185] Key type lgssc registered [12205.716077] Lustre: Echo OBD driver; http://www.lustre.org/ [12273.551774] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [12288.048133] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12288.123276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12289.729963] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12289.809046] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12290.204769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12297.851600] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12309.805910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12309.991943] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12310.054887] Lustre: 297692:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12310.576136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12316.413439] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12324.351389] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12330.685695] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12355.003943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12355.366579] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12355.647614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12364.220748] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12372.989952] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12378.088903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12378.109539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12378.131036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12378.151132] Lustre: Skipped 1 previous similar message [12383.209215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12383.224172] Lustre: Skipped 1 previous similar message [12388.351534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12388.373581] Lustre: Skipped 1 previous similar message [12389.856478] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12390.307158] Lustre: server umount lustre-OST0000 complete [12397.538653] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12397.547192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12397.573756] Lustre: Skipped 1 previous similar message [12397.587751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12398.569627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12403.438284] Lustre: server umount lustre-MDT0000 complete [12408.809353] LustreError: 297702:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12408.876249] LustreError: 297702:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12410.325240] LustreError: 297682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779705395 with bad export cookie 13299661797671057707 [12410.349460] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12410.390514] LustreError: 297682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12411.320270] Lustre: server umount lustre-MDT0001 complete [12431.466824] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [12435.816944] Key type lgssc unregistered [12436.301526] LNet: 300936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12436.310824] LNetError: 300936:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12436.352664] LNet: Removed LNI 192.168.201.157@tcp [12437.744672] Key type .llcrypt unregistered [12437.751545] Key type ._llcrypt unregistered [12471.268160] Key type ._llcrypt registered [12471.280477] Key type .llcrypt registered [12471.511882] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [12473.373540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12473.469535] alg: No test for adler32 (adler32-zlib) [12475.462725] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [12475.774228] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [12477.561621] Key type lgssc registered [12479.945904] Lustre: Echo OBD driver; http://www.lustre.org/ [12491.407816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12491.438573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12493.148937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12499.980583] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12503.523433] LustreError: 302338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12503.567782] LustreError: 302338:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12508.645497] LustreError: 302339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12512.330201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12512.853782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12519.766806] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12528.462025] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12535.138542] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12546.287714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12547.067465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12555.943985] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12565.008427] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12578.257436] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.57@tcp (stopping) [12579.322316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12579.334464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12579.360315] LustreError: 302339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12579.504531] Lustre: server umount lustre-MDT0000 complete [12583.320507] LustreError: 302334:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12583.351645] LustreError: 302334:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12584.347061] LustreError: 302319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779705569 with bad export cookie 7474105940141145517 [12584.349373] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12584.364343] LustreError: 302319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12584.944219] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12584.960445] Lustre: Skipped 2 previous similar messages [12584.971634] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12588.439652] LustreError: 303241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12588.480815] LustreError: 303241:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12588.504402] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.57@tcp (stopping) [12591.115224] Lustre: server umount lustre-MDT0001 complete [12600.846672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12613.135275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12628.964154] LustreError: 301648:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001425fb91 x1866156432460416/t0(0) o250->MGC192.168.201.157@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12629.466923] LustreError: 303947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12629.501046] LustreError: 303947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12629.641070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12629.784177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [12638.169738] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12644.368910] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12649.467766] LustreError: 303948:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12649.503029] LustreError: 303948:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [12651.816627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12652.681323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12658.184596] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12661.022414] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12669.712555] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12672.662407] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12678.831993] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12681.885443] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [12690.913781] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12690.932684] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12690.951388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12690.961879] Lustre: Skipped 1 previous similar message [12693.477623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12693.494417] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12695.309614] Lustre: server umount lustre-OST0000 complete [12703.714111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12703.727472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12703.759322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12703.783103] Lustre: Skipped 2 previous similar messages [12707.923771] Lustre: server umount lustre-MDT0000 complete [12709.346169] LustreError: 305601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12709.381238] LustreError: 305601:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12713.371555] LustreError: 302319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779705698 with bad export cookie 7474105940141152664 [12713.395521] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12713.406954] LustreError: 302319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12714.025144] Lustre: server umount lustre-MDT0001 complete [12730.100681] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [12734.083765] Key type lgssc unregistered [12734.707234] LNet: 308033:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12734.718264] LNetError: 308033:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12734.739741] LNet: Removed LNI 192.168.201.157@tcp [12736.092257] Key type .llcrypt unregistered [12736.096829] Key type ._llcrypt unregistered [12769.944417] Key type ._llcrypt registered [12769.947889] Key type .llcrypt registered [12770.082824] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [12771.512049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12771.789390] alg: No test for adler32 (adler32-zlib) [12773.117425] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [12773.383711] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [12775.137129] Key type lgssc registered [12776.769711] Lustre: Echo OBD driver; http://www.lustre.org/ [12786.898864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12786.931867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12789.200699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12796.975361] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12799.464338] LustreError: 309442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12799.488872] LustreError: 309442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12804.582594] LustreError: 309443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12809.702413] LustreError: 309442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12809.955829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12810.659687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12817.206400] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12825.712669] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12832.618372] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12840.419107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12840.427266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12840.446737] Lustre: Skipped 1 previous similar message [12845.542665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12845.555341] Lustre: Skipped 2 previous similar messages [12850.661714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12850.690214] Lustre: Skipped 1 previous similar message [12852.192162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12852.587570] Lustre: server umount lustre-MDT0000 complete [12855.781404] LustreError: 309439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12855.830232] LustreError: 309439:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12858.049395] LustreError: 309422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779705843 with bad export cookie 7216657626391121702 [12858.057714] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12858.088326] LustreError: 309422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12858.723267] Lustre: server umount lustre-MDT0001 complete [12878.335759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12878.903664] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12879.012877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12879.412376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12885.050431] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12897.314903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12897.446421] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12897.510387] Lustre: 311811:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12897.905490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12904.753123] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12913.276258] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12919.143962] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12932.421597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12932.823799] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [12932.861422] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [12932.893171] LustreError: 313470:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [12932.912970] LustreError: 313470:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [12932.932584] LustreError: 313470:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [12933.105420] Lustre: server umount lustre-OST0000 complete [12933.119984] LustreError: 313470:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [12949.580673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12949.940968] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12950.274027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12956.006923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [12960.576709] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12969.614855] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12980.970811] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12981.108732] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [12981.184896] Lustre: lustre-OST0001: new disk, initializing [12981.187817] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12981.194555] Lustre: 314901:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12981.301148] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12989.676650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12989.710214] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12989.843933] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12990.527646] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12999.832611] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [13005.288339] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13005.308367] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13005.318798] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [13005.339877] Lustre: Skipped 1 previous similar message [13005.360231] Lustre: Skipped 1 previous similar message [13010.415665] LustreError: 313937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13010.433333] LustreError: 313937:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13010.486229] Lustre: server umount lustre-OST0001 complete [13017.573049] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13017.579042] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13017.588674] LustreError: Skipped 2 previous similar messages [13017.611692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13020.668233] LustreError: 313938:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13020.707057] LustreError: 313938:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [13025.788114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13025.802426] Lustre: Skipped 4 previous similar messages [13030.371681] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13030.596461] Lustre: server umount lustre-OST0000 complete [13041.130240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13041.153196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13041.183619] Lustre: Skipped 1 previous similar message [13042.926878] Lustre: server umount lustre-MDT0000 complete [13046.772885] LustreError: 311821:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13046.815860] LustreError: 311821:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [13048.625993] LustreError: 311802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779706033 with bad export cookie 7216657626391122255 [13048.646531] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13048.655734] LustreError: 311802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13049.262831] Lustre: server umount lustre-MDT0001 complete [13065.818231] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [13069.613746] Key type lgssc unregistered [13069.980398] LNet: 316734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13069.995505] LNetError: 316734:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13070.023847] LNet: Removed LNI 192.168.201.157@tcp [13071.209598] Key type .llcrypt unregistered [13071.212445] Key type ._llcrypt unregistered [13166.438088] Key type ._llcrypt registered [13166.443109] Key type .llcrypt registered [13166.615579] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [13188.796203] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [13190.655460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13190.768782] alg: No test for adler32 (adler32-zlib) [13192.058741] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [13192.388331] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [13194.120527] Key type lgssc registered [13195.780823] Lustre: Echo OBD driver; http://www.lustre.org/ [13256.393319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13256.440054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13257.788280] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13257.835756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13257.922887] Lustre: lustre-MDT0000: new disk, initializing [13258.037323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13258.050733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13265.192497] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13281.277719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13281.406787] Lustre: 321646:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13281.457232] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13281.465399] Lustre: Skipped 1 previous similar message [13281.533152] Lustre: lustre-MDT0001: new disk, initializing [13281.643192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13281.697656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13281.708172] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13288.252901] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13298.975703] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13305.428767] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13315.827806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13316.382265] Lustre: lustre-OST0000: new disk, initializing [13316.394882] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13316.407502] Lustre: 323388:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13316.564404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13323.870019] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13323.882592] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13324.009987] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13325.305427] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13335.919026] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13342.676286] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13343.025754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13350.169939] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13350.589266] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13354.978044] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13354.981051] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13354.985921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13355.014940] Lustre: Skipped 1 previous similar message [13359.823242] Lustre: server umount lustre-OST0000 complete [13365.181183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13365.219639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13365.227031] Lustre: Skipped 1 previous similar message [13370.338924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13370.352103] Lustre: Skipped 1 previous similar message [13371.713027] Lustre: server umount lustre-MDT0000 complete [13377.322220] LustreError: 323387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779706362 with bad export cookie 9450407516787252784 [13377.322945] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13377.355941] LustreError: 323387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13377.827422] Lustre: server umount lustre-MDT0001 complete [13390.244399] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 06:52:52 (1779706372) [13402.220577] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 06:53:04 (1779706384) [13415.246491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13416.077514] LustreError: 325999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13416.116528] LustreError: 325999:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [13416.260262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13421.538284] LustreError: 326000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13423.421416] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13426.659847] LustreError: 325999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13431.781162] LustreError: 326000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13435.300316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13435.814337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13442.412440] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13452.079503] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13458.407326] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13468.896621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13469.575941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13481.389466] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13491.639400] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13502.574286] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [13506.376313] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [13508.925690] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [13511.714450] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [13513.875156] Lustre: Failing over lustre-MDT0000 [13514.170825] Lustre: server umount lustre-MDT0000 complete [13515.214384] LustreError: 325995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13516.773496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13516.805185] Lustre: Skipped 3 previous similar messages [13525.412473] LustreError: 325994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13525.443688] LustreError: 325994:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [13532.132548] Lustre: 319867:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779706501/real 1779706501] req@0000000093cb098e x1866157184334336/t0(0) o400->MGC192.168.201.157@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1779706517 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13532.188798] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13536.542730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13541.785131] LustreError: 327029:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13541.833794] LustreError: 327029:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [13542.372585] LustreError: 319865:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004e2a6977 x1866157184339712/t0(0) o250->MGC192.168.201.157@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13542.875993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13542.965116] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [13546.907606] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [13548.025661] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [13548.030234] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13548.068313] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [13548.174145] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13559.950677] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13563.431518] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13567.860192] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [13570.825966] Lustre: Failing over lustre-MDT0000 [13571.188974] Lustre: server umount lustre-MDT0000 complete [13573.606800] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13573.613954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13577.627073] LustreError: 325994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13577.663651] LustreError: 325994:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [13588.961055] Lustre: 319866:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779706558/real 1779706558] req@000000000ac8172d x1866157184356736/t0(0) o400->MGC192.168.201.157@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1779706574 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13589.001859] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13594.167620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13599.207500] LustreError: 319865:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000020b6b5e5 x1866157184362240/t0(0) o250->MGC192.168.201.157@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13599.757384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13599.818424] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [13604.261967] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [13604.876565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [13604.899372] Lustre: Skipped 2 previous similar messages [13604.943973] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [13605.044884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13606.290966] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13618.905625] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13621.685934] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13630.949117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13630.955789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13630.990949] Lustre: Skipped 2 previous similar messages [13631.001392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13635.557978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13635.572451] Lustre: Skipped 2 previous similar messages [13637.387939] Lustre: server umount lustre-MDT0000 complete [13643.137412] LustreError: 325980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779706628 with bad export cookie 9450407516787255500 [13643.160367] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13643.173373] LustreError: 325980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13643.765120] Lustre: server umount lustre-MDT0001 complete [13660.693860] Lustre: server umount lustre-OST0000 complete [13685.347672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13686.108119] LustreError: 331784:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13686.144629] LustreError: 331784:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [13686.282468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13692.786808] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13705.640299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13712.097645] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13720.540833] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13727.442717] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13738.143402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13748.954854] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13757.294131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13757.617740] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13766.633993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13766.653354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13766.677414] Lustre: Skipped 2 previous similar messages [13766.696073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13771.612229] Lustre: server umount lustre-OST0000 complete [13777.895899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13777.917081] LustreError: Skipped 1 previous similar message [13783.014396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13783.032085] Lustre: Skipped 5 previous similar messages [13786.394961] Lustre: server umount lustre-MDT0000 complete [13791.890401] LustreError: 331766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779706777 with bad export cookie 9450407516787256242 [13791.894992] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13791.911876] LustreError: 331766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13792.453980] Lustre: server umount lustre-MDT0001 complete [13811.559431] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [13815.805758] Key type lgssc unregistered [13816.252978] LNet: 334986:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13816.272393] LNetError: 334986:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13816.301880] LNet: Removed LNI 192.168.201.157@tcp [13817.637718] Key type .llcrypt unregistered [13817.642752] Key type ._llcrypt unregistered [13854.337700] Key type ._llcrypt registered [13854.340157] Key type .llcrypt registered [13854.467813] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [13856.467746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13856.514232] alg: No test for adler32 (adler32-zlib) [13857.756895] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [13858.190627] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [13859.952904] Key type lgssc registered [13861.711981] Lustre: Echo OBD driver; http://www.lustre.org/ [13873.487330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13873.525251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13875.523747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13882.461279] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13884.913980] LustreError: 336494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13884.945450] LustreError: 336494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13890.034418] LustreError: 336493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13893.918261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13894.556698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13902.636491] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13913.745971] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13919.556696] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13930.311636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13931.075894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13940.293854] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13945.503679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13948.939607] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13966.307780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13966.318194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13966.334940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13966.379948] Lustre: Skipped 2 previous similar messages [13971.433264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13971.453684] Lustre: Skipped 4 previous similar messages [13976.546022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13976.558024] Lustre: Skipped 2 previous similar messages [13979.105926] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13979.407384] Lustre: server umount lustre-MDT0000 complete [13981.690249] LustreError: 336488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13981.726623] LustreError: 336488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13984.577117] LustreError: 337225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779706969 with bad export cookie 15931697313118611153 [13984.581204] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13984.595862] LustreError: 337225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13984.979284] Lustre: server umount lustre-MDT0001 complete [14000.961697] Lustre: server umount lustre-OST0000 complete [14019.499900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14020.274774] LustreError: 339907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14020.396846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14025.700459] LustreError: 339908:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14026.575625] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [14035.943671] LustreError: 339907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14035.972069] LustreError: 339907:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14039.659023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14040.193274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14047.088676] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [14057.121891] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14063.758896] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14074.358630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14074.834900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14084.051637] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [14090.191611] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [14092.176762] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14110.707846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14110.709701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14110.745531] Lustre: Skipped 1 previous similar message [14113.269893] Lustre: server umount lustre-MDT0000 complete [14115.811185] LustreError: 341684:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14115.833517] LustreError: 341684:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [14118.354482] LustreError: 339887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779707103 with bad export cookie 15931697313118612266 [14118.359309] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14118.369028] LustreError: 339887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14118.916541] Lustre: server umount lustre-MDT0001 complete [14135.080921] Lustre: server umount lustre-OST0000 complete [14156.831565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14157.332484] LustreError: 343482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14157.422620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14163.259648] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [14175.811894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14182.748377] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [14190.935243] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14196.469688] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14204.987461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14205.558250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14205.577260] Lustre: Skipped 1 previous similar message [14213.441391] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [14220.585317] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14227.070153] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [14241.769694] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14241.782921] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14241.809480] Lustre: Skipped 1 previous similar message [14241.819108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14241.834389] Lustre: Skipped 2 previous similar messages [14242.374555] Lustre: server umount lustre-OST0000 complete [14247.908270] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14247.917751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14253.029591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14253.038042] Lustre: Skipped 3 previous similar messages [14257.150533] LustreError: 343479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14257.177283] LustreError: 343479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [14257.399915] Lustre: server umount lustre-MDT0000 complete [14262.523914] LustreError: 343462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779707247 with bad export cookie 15931697313118613288 [14262.528921] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14262.539347] LustreError: 343462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14263.150179] Lustre: server umount lustre-MDT0001 complete [14278.772983] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [14282.397294] Key type lgssc unregistered [14282.700946] LNet: 346687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14282.709961] LNetError: 346687:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14282.738714] LNet: Removed LNI 192.168.201.157@tcp [14283.799158] Key type .llcrypt unregistered [14283.804951] Key type ._llcrypt unregistered [14319.305313] Key type ._llcrypt registered [14319.312659] Key type .llcrypt registered [14319.482693] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [14321.002750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14321.129618] alg: No test for adler32 (adler32-zlib) [14322.360718] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [14322.756875] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [14324.680350] Key type lgssc registered [14326.562410] Lustre: Echo OBD driver; http://www.lustre.org/ [14339.395623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14339.435185] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14339.447206] LustreError: 348114:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [14339.482407] LustreError: 348114:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [14339.494631] LustreError: 348114:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [14339.503857] LustreError: 348114:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [14339.521704] LustreError: 348114:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [14351.888670] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [14351.895565] LustreError: 348512:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [14351.916590] LustreError: 348512:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [14351.928940] LustreError: 348512:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [14351.940092] LustreError: 348512:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [14351.954564] LustreError: 348512:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [14380.975171] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [14384.441718] Key type lgssc unregistered [14384.729976] LNet: 349500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14384.735838] LNetError: 349500:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14384.756598] LNet: Removed LNI 192.168.201.157@tcp [14385.792287] Key type .llcrypt unregistered [14385.797886] Key type ._llcrypt unregistered [14408.846963] Key type ._llcrypt registered [14408.853717] Key type .llcrypt registered [14409.044068] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [14429.003670] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [14431.544334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14431.659662] alg: No test for adler32 (adler32-zlib) [14432.935328] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [14433.371770] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [14435.144406] Key type lgssc registered [14437.274275] Lustre: Echo OBD driver; http://www.lustre.org/ [14488.029771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14488.075678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14489.596661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14489.675045] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14489.837555] Lustre: lustre-MDT0000: new disk, initializing [14489.996964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14490.077088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14496.838776] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [14513.175578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14513.321950] Lustre: 352625:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14513.374994] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14513.379168] Lustre: Skipped 1 previous similar message [14513.443634] Lustre: lustre-MDT0001: new disk, initializing [14513.592479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14513.673423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14513.703753] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14520.741565] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [14532.946328] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14539.189163] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14550.711807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14551.344826] Lustre: lustre-OST0000: new disk, initializing [14551.350215] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14551.360587] Lustre: 354372:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14551.563413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14560.373132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14560.394690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14560.586482] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14561.925602] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [14572.742329] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14579.680144] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14580.111286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14587.808412] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14588.359864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14596.589396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14596.590977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14596.604734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14596.648539] Lustre: Skipped 1 previous similar message [14601.700666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14601.713633] Lustre: Skipped 2 previous similar messages [14606.304290] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14606.649556] Lustre: server umount lustre-OST0000 complete [14612.859033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14612.864041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14612.880218] Lustre: Skipped 1 previous similar message [14612.897744] Lustre: Skipped 1 previous similar message [14618.083382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14618.107254] Lustre: Skipped 1 previous similar message [14623.202893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14623.211091] Lustre: Skipped 1 previous similar message [14627.296172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14627.542154] Lustre: server umount lustre-MDT0000 complete [14633.424571] LustreError: 354371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779707618 with bad export cookie 142620949271009148 [14633.447506] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14633.449407] LustreError: 354371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14633.989128] Lustre: server umount lustre-MDT0001 complete [14648.214788] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [14651.303720] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 07:13:53 (1779707633) [14661.362092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14661.816424] LustreError: 356693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14661.839107] LustreError: 356693:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [14661.910790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14667.234467] LustreError: 356694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14668.560051] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [14671.333884] LustreError: 356693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14676.460201] LustreError: 356694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14681.216464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14681.581582] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [14681.600729] Lustre: Skipped 2 previous similar messages [14681.787766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14688.426244] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [14696.439832] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14702.075052] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14711.727057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14712.264970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14721.236481] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [14730.472681] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14740.224388] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14740.440680] Lustre: lustre-OST0001: new disk, initializing [14740.448681] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [14740.459772] Lustre: 359195:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14740.648646] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14747.229536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14747.239712] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14747.405527] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14748.852083] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [14759.412242] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [14766.528132] Lustre: lustre-OST0001: Not available for connect from 192.168.201.57@tcp (stopping) [14768.097259] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14768.114086] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14768.120038] LustreError: Skipped 1 previous similar message [14768.140485] Lustre: Skipped 1 previous similar message [14770.694887] Lustre: server umount lustre-OST0001 complete [14771.616111] LustreError: 358287:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14783.483492] LustreError: 358845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14783.515102] LustreError: 358845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [14793.187971] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14793.192112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14793.197380] LustreError: Skipped 1 previous similar message [14793.223592] Lustre: Skipped 1 previous similar message [14796.822219] Lustre: server umount lustre-OST0000 complete [14803.189881] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14803.230956] Lustre: Skipped 1 previous similar message [14803.232315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14803.262736] Lustre: Skipped 5 previous similar messages [14809.693443] Lustre: server umount lustre-MDT0000 complete [14813.666795] LustreError: 356694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14813.699955] LustreError: 356694:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [14815.186953] LustreError: 359194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779707800 with bad export cookie 142620949271010030 [14815.190190] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14815.203746] LustreError: 359194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14821.649402] Lustre: server umount lustre-MDT0001 complete [14840.983280] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [14845.392672] Key type lgssc unregistered [14845.766248] LNet: 361075:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14845.798545] LNetError: 361075:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14845.832896] LNet: Removed LNI 192.168.201.157@tcp [14847.038530] Key type .llcrypt unregistered [14847.041431] Key type ._llcrypt unregistered [14928.132334] Key type ._llcrypt registered [14928.136302] Key type .llcrypt registered [14928.281972] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [14929.949244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14930.051473] alg: No test for adler32 (adler32-zlib) [14931.495549] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [14931.855726] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [14933.696305] Key type lgssc registered [14935.503197] Lustre: Echo OBD driver; http://www.lustre.org/ [14946.641182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14946.699482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14948.226650] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [14948.264784] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14948.524462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14954.944648] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [14967.603422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14967.803798] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [14967.909867] Lustre: 363865:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14968.436263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14974.374559] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [14982.874195] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14989.667917] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14993.889343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14993.900418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14993.925840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14995.942456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14995.961546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14999.415716] Lustre: server umount lustre-MDT0000 complete [15005.156712] LustreError: 363856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779707990 with bad export cookie 18338852995264850695 [15005.160568] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15005.188638] LustreError: 363856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15005.966524] Lustre: server umount lustre-MDT0001 complete [15021.225930] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 07:20:03 (1779708003) [15030.946539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15031.823512] LustreError: 366155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15031.860916] LustreError: 366155:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [15031.931548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15036.901856] LustreError: 366156:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15038.536568] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [15042.019576] LustreError: 366155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15047.138104] LustreError: 366156:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15051.099408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15051.804766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15059.143427] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [15067.904583] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15073.708777] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15084.854212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15085.322604] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [15085.687085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15095.849710] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [15107.905955] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15118.947632] Lustre: Permanently deactivating lustre-OST0000 [15119.000235] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [15127.687025] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [15130.331889] LustreError: 368758:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [15130.339430] LustreError: 368758:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [15130.347929] LustreError: 368758:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [15137.801227] LustreError: 368807:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [15137.814195] LustreError: 368807:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [15146.811283] Lustre: server umount lustre-OST0000 complete [15150.868193] LustreError: 369057:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [15150.884504] LustreError: 369057:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [15154.147068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15154.170790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15154.197883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15154.204243] Lustre: Skipped 1 previous similar message [15158.760401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15158.765032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15158.808894] Lustre: Skipped 1 previous similar message [15163.874979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15163.895487] Lustre: Skipped 1 previous similar message [15165.084605] Lustre: server umount lustre-MDT0000 complete [15169.006741] LustreError: 366155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15169.051806] LustreError: 366155:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [15170.615466] LustreError: 366872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779708155 with bad export cookie 18338852995264851276 [15170.621946] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15170.633185] LustreError: 366872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15180.514599] Lustre: server umount lustre-MDT0001 complete [15194.413092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15201.291850] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [15205.349832] LustreError: 370201:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [15215.631909] Lustre: 370540:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.157@tcp [15219.133171] Lustre: server umount MGS complete [15237.811418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15238.521339] LustreError: 371267:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15238.557319] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [15238.562061] Lustre: Skipped 1 previous similar message [15238.607386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15245.403844] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [15259.122879] LustreError: 371268:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15259.147512] LustreError: 371268:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [15259.679884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15260.142687] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [15260.233337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15267.628205] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [15277.366734] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15283.675354] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15286.891971] Lustre: Permanently reactivating lustre-OST0000 [15286.942773] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [15286.946930] Lustre: Skipped 2 previous similar messages [15313.890222] LustreError: 372775:0:(osp_dev.c:693:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [15325.177138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15325.536156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15334.671949] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [15344.927762] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15354.350766] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15354.365098] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15354.403538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15360.061169] Lustre: server umount lustre-OST0000 complete [15367.667328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15367.690029] LustreError: Skipped 1 previous similar message [15367.695399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15367.743370] Lustre: Skipped 1 previous similar message [15370.727429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15370.732667] Lustre: Skipped 4 previous similar messages [15372.146155] Lustre: server umount lustre-MDT0000 complete [15375.864880] LustreError: 371263:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15375.880722] LustreError: 371263:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [15377.234721] LustreError: 371247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779708362 with bad export cookie 18338852995264852326 [15377.240040] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15377.245271] LustreError: 371247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15377.571752] Lustre: server umount lustre-MDT0001 complete [15395.439296] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [15399.121435] Key type lgssc unregistered [15399.529995] LNet: 375151:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15399.552687] LNetError: 375151:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15399.591449] LNet: Removed LNI 192.168.201.157@tcp [15400.879827] Key type .llcrypt unregistered [15400.883899] Key type ._llcrypt unregistered [15429.729441] Key type ._llcrypt registered [15429.745204] Key type .llcrypt registered [15429.900186] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [15454.191879] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [15456.315603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15456.691854] alg: No test for adler32 (adler32-zlib) [15457.889843] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [15458.397422] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [15460.272515] Key type lgssc registered [15461.519236] Lustre: Echo OBD driver; http://www.lustre.org/ [15517.231950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15517.296769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15518.981440] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15519.053684] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15519.184327] Lustre: lustre-MDT0000: new disk, initializing [15519.374676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15519.431366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15527.379676] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [15545.588296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15545.841905] Lustre: 378282:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15546.027350] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15546.040448] Lustre: Skipped 1 previous similar message [15546.238509] Lustre: lustre-MDT0001: new disk, initializing [15546.398839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15546.504608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15546.534841] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15554.615907] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [15567.869513] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15574.794372] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15584.277481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15584.632725] Lustre: lustre-OST0000: new disk, initializing [15584.637986] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15584.644833] Lustre: 380027:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15584.792310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15592.615017] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15592.635505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15592.775757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15594.212449] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [15605.373770] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15613.677503] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15614.099531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15621.814876] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15622.122941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15628.771698] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15628.778674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15628.794303] LustreError: Skipped 1 previous similar message [15628.798499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15628.820662] Lustre: Skipped 1 previous similar message [15628.846086] Lustre: Skipped 1 previous similar message [15631.539727] Lustre: server umount lustre-OST0000 complete [15638.498180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15638.517215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15638.558940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15643.337214] Lustre: server umount lustre-MDT0000 complete [15649.017893] LustreError: 378275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779708634 with bad export cookie 6094953125861601306 [15649.018777] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15649.034033] LustreError: 378275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15649.716294] Lustre: server umount lustre-MDT0001 complete [15664.540194] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [15668.016232] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 07:30:49 (1779708649) [15679.505856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15680.284303] LustreError: 382345:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15680.325196] LustreError: 382345:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [15680.417214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15685.601181] LustreError: 382346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15688.069291] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [15690.721876] LustreError: 382345:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15695.845852] LustreError: 382346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15700.991531] LustreError: 382345:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15703.798437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15704.817093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15712.913637] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [15724.947203] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15732.324481] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15744.073806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15744.767555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15754.218581] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [15763.243868] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15765.987891] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [15769.993386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [15780.844720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15780.845476] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15780.857184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15780.857203] Lustre: Skipped 2 previous similar messages [15780.862437] LustreError: Skipped 1 previous similar message [15780.896354] Lustre: Skipped 2 previous similar messages [15784.871729] Lustre: server umount lustre-OST0000 complete [15791.074780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15791.084588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15791.123109] Lustre: Skipped 1 previous similar message [15791.129363] Lustre: Skipped 1 previous similar message [15796.194594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15796.210492] Lustre: Skipped 2 previous similar messages [15796.729982] Lustre: server umount lustre-MDT0000 complete [15801.313857] LustreError: 382346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15801.360918] LustreError: 382346:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [15802.697979] LustreError: 382325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779708787 with bad export cookie 6094953125861602188 [15802.711023] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15802.723049] LustreError: 382325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15803.371943] Lustre: server umount lustre-MDT0001 complete [15824.032514] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [15828.819476] Key type lgssc unregistered [15829.402811] LNet: 385748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15829.419807] LNetError: 385748:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15829.466645] LNet: Removed LNI 192.168.201.157@tcp [15831.060978] Key type .llcrypt unregistered [15831.067244] Key type ._llcrypt unregistered [15866.043496] Key type ._llcrypt registered [15866.047683] Key type .llcrypt registered [15866.176817] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [15866.658454] Key type .llcrypt unregistered [15866.667350] Key type ._llcrypt unregistered [15892.828555] Key type ._llcrypt registered [15892.831528] Key type .llcrypt registered [15893.020640] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [15895.243232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15895.415288] alg: No test for adler32 (adler32-zlib) [15896.848529] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [15897.377367] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [15899.131667] Key type lgssc registered [15901.244200] Lustre: Echo OBD driver; http://www.lustre.org/ [15914.139145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15914.167386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15916.112038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15922.751142] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [15926.249842] LustreError: 387935:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15926.283399] LustreError: 387935:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [15931.368929] LustreError: 387934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15935.398680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15935.882632] LustreError: 387935:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15935.928726] LustreError: 387935:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [15935.944253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15941.048854] LustreError: 387934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15941.090065] LustreError: 387934:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [15941.098606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [15946.211152] LustreError: 387931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15946.242949] LustreError: 387931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [15946.596608] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [15954.597401] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15956.459925] LustreError: 387935:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15956.492390] LustreError: 387935:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [15964.617421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15965.139514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15971.600856] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [15985.636409] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15985.637930] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15985.661712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15990.145159] Lustre: server umount lustre-OST0000 complete [15995.875454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15995.877387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15995.919194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15995.925832] Lustre: Skipped 1 previous similar message [15995.933587] Lustre: Skipped 2 previous similar messages [16001.012756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16001.030041] Lustre: Skipped 1 previous similar message [16001.948451] Lustre: server umount lustre-MDT0000 complete [16005.098184] LustreError: 388660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16005.128936] LustreError: 388660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [16007.876604] LustreError: 387913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779708993 with bad export cookie 1731307826282699147 [16007.880667] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16007.897472] LustreError: 387913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16008.271852] Lustre: server umount lustre-MDT0001 complete [16026.813881] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [16031.128935] Key type lgssc unregistered [16031.766188] LNet: 390943:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16031.784664] LNetError: 390943:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16031.801190] LNet: Removed LNI 192.168.201.157@tcp [16033.192849] Key type .llcrypt unregistered [16033.194901] Key type ._llcrypt unregistered [16068.193636] Key type ._llcrypt registered [16068.197561] Key type .llcrypt registered [16068.323833] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [16070.283920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16070.691379] alg: No test for adler32 (adler32-zlib) [16071.848832] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [16072.518576] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [16074.280168] Key type lgssc registered [16076.936349] Lustre: Echo OBD driver; http://www.lustre.org/ [16088.945350] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16088.963833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16104.481447] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [16109.602215] LustreError: 392318:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.157@tcp: failed processing log, type 4: rc = -110 [16136.480339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16146.611567] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [16155.549324] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16165.329414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16166.417298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16166.545510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [16171.496351] LustreError: 392344:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16171.512219] LustreError: 392344:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [16172.522984] LustreError: 393245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16172.543510] LustreError: 393245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [16173.613538] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [16177.655194] LustreError: 393246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16177.686392] LustreError: 393246:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [16182.761600] LustreError: 393246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16182.821251] LustreError: 393246:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [16187.897970] LustreError: 393245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16187.936914] LustreError: 393245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [16189.153035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16189.846085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16197.566604] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [16206.678039] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16213.172420] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16222.690219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16222.703194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16222.723327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16225.773512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16225.775163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16225.775189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16225.858362] Lustre: Skipped 1 previous similar message [16229.025055] Lustre: server umount lustre-OST0000 complete [16236.005684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16236.008893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16236.018366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16236.043398] Lustre: Skipped 1 previous similar message [16236.078434] Lustre: Skipped 1 previous similar message [16241.121660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16241.140306] Lustre: Skipped 1 previous similar message [16246.246524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16246.266043] Lustre: Skipped 1 previous similar message [16249.313053] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16249.745589] Lustre: server umount lustre-MDT0000 complete [16251.377672] LustreError: 393247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16251.413296] LustreError: 393247:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [16255.951730] LustreError: 392325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779709241 with bad export cookie 15274283577665555562 [16255.960129] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16255.972879] LustreError: 392325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16256.787064] Lustre: server umount lustre-MDT0001 complete [16276.063194] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [16280.721915] Key type lgssc unregistered [16281.276697] LNet: 395555:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16281.296315] LNetError: 395555:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16281.319351] LNet: Removed LNI 192.168.201.157@tcp [16282.704293] Key type .llcrypt unregistered [16282.713862] Key type ._llcrypt unregistered [16325.952751] Key type ._llcrypt registered [16325.961182] Key type .llcrypt registered [16326.197207] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [16328.984505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16329.077960] alg: No test for adler32 (adler32-zlib) [16331.277822] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [16331.693184] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [16333.688185] Key type lgssc registered [16336.148175] Lustre: Echo OBD driver; http://www.lustre.org/ [16349.855431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16349.879836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16351.682943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16359.220131] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [16360.953909] LustreError: 396961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16361.031835] LustreError: 396961:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [16366.058588] LustreError: 396960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16371.179634] LustreError: 396961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16372.985503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16373.646993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16380.261151] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [16389.476905] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16395.923258] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16407.796405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16408.637454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16418.869301] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [16420.990109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [16427.448432] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16435.172050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16435.190640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16435.204167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16436.709162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16436.710604] LustreError: 397674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16436.798172] Lustre: server umount lustre-MDT0000 complete [16440.671209] LustreError: 396940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.57@tcp arrived at 1779709425 with bad export cookie 7058363088899287430 [16440.698434] LustreError: 396940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16441.840893] LustreError: 397674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16441.892203] LustreError: 397674:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [16444.365691] LustreError: 396941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779709429 with bad export cookie 7058363088899286534 [16444.374631] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16444.384750] LustreError: 396941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16444.391559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16444.391591] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16444.391594] Lustre: Skipped 1 previous similar message [16444.394758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16444.471724] Lustre: Skipped 1 previous similar message [16446.955897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16449.954824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16449.959551] LustreError: 397674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16450.022658] LustreError: 397674:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [16450.939342] Lustre: server umount lustre-OST0000 complete [16461.946365] Lustre: server umount lustre-MDT0001 complete [16481.653878] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [16485.512762] Key type lgssc unregistered [16485.970903] LNet: 400316:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16485.987904] LNetError: 400316:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16486.033651] LNet: Removed LNI 192.168.201.157@tcp [16487.161488] Key type .llcrypt unregistered [16487.168417] Key type ._llcrypt unregistered [16526.139867] Key type ._llcrypt registered [16526.143266] Key type .llcrypt registered [16526.386524] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [16528.248925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16528.397058] alg: No test for adler32 (adler32-zlib) [16529.734307] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [16530.217861] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [16531.969805] Key type lgssc registered [16533.888598] Lustre: Echo OBD driver; http://www.lustre.org/ [16545.992093] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16546.015703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16547.812581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16553.551540] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [16557.102505] LustreError: 401719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16557.135939] LustreError: 401719:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [16562.145593] LustreError: 401718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16565.331818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16565.988285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16572.846221] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [16582.248572] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16589.809927] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16600.425857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16601.293685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16611.127658] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [16617.652298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [16618.645811] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16626.627939] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.57@tcp (stopping) [16627.565622] Lustre: server umount lustre-MDT0001 complete [16627.706694] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [16627.716886] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16627.740296] LustreError: 402429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16633.327733] LustreError: 401714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16633.368512] LustreError: 401714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [16638.443175] LustreError: 402429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16638.464701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16638.500736] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16638.535759] Lustre: Skipped 1 previous similar message [16638.551769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16643.555877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16644.684349] Lustre: server umount lustre-OST0000 complete [16648.688660] LustreError: 402429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16648.723820] LustreError: 402429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [16651.276759] Lustre: server umount lustre-MDT0000 complete [16671.924712] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [16676.357628] Key type lgssc unregistered [16676.912894] LNet: 405052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16676.932846] LNetError: 405052:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16676.972115] LNet: Removed LNI 192.168.201.157@tcp [16678.534536] Key type .llcrypt unregistered [16678.537772] Key type ._llcrypt unregistered [16716.217761] Key type ._llcrypt registered [16716.221381] Key type .llcrypt registered [16716.339555] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [16740.274934] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [16742.702164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16742.743397] alg: No test for adler32 (adler32-zlib) [16744.759158] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [16745.432148] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [16747.256516] Key type lgssc registered [16749.720557] Lustre: Echo OBD driver; http://www.lustre.org/ [16807.434458] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [16831.031318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16831.076685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16832.529714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16832.591313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16832.742444] Lustre: lustre-MDT0000: new disk, initializing [16832.854505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16832.876317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16838.841717] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [16862.500640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16862.797730] Lustre: lustre-OST0000: new disk, initializing [16862.802855] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16862.809476] Lustre: Skipped 1 previous similar message [16862.817534] Lustre: 410044:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16862.899891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16870.455964] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [16871.576115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16871.589111] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16871.651595] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16881.355590] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16901.444382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16901.612785] Lustre: 409041:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16901.693115] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16901.784575] Lustre: lustre-MDT0001: new disk, initializing [16901.962045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16902.025656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [16902.051887] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [16907.926843] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [16921.060281] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [16921.374354] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [16928.492158] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [16928.888638] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [16937.426312] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.57@tcp (stopping) [16938.980626] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [16938.982857] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16939.018487] Lustre: Skipped 1 previous similar message [16939.019898] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [16939.031388] Lustre: Skipped 1 previous similar message [16942.495440] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.57@tcp (stopping) [16943.162186] Lustre: server umount lustre-MDT0001 complete [16949.228253] LustreError: 409053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16949.271128] LustreError: 409053:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [16954.341832] LustreError: 409048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16954.361291] LustreError: 409048:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [16959.486602] LustreError: 410645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16959.530346] LustreError: 410645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [16964.067356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16964.075131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16964.113724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16964.584723] LustreError: 411510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16969.701708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16969.705442] LustreError: 409052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16970.164799] Lustre: server umount lustre-OST0000 complete [16990.176509] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16990.874813] Lustre: server umount lustre-MDT0000 complete [17017.195237] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [17021.776757] Key type lgssc unregistered [17022.229798] LNet: 413639:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17022.248971] LNetError: 413639:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [17022.293850] LNet: Removed LNI 192.168.201.157@tcp [17023.571636] Key type .llcrypt unregistered [17023.574604] Key type ._llcrypt unregistered [17089.841655] Key type ._llcrypt registered [17089.844254] Key type .llcrypt registered [17089.990219] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [17090.354347] Key type .llcrypt unregistered [17090.358388] Key type ._llcrypt unregistered [17120.947048] Key type ._llcrypt registered [17120.950928] Key type .llcrypt registered [17121.157622] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [17124.088295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17124.121376] alg: No test for adler32 (adler32-zlib) [17125.465728] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [17125.856700] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [17127.616929] Key type lgssc registered [17128.983456] Lustre: Echo OBD driver; http://www.lustre.org/ [17167.917959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17167.993372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17169.695334] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17169.766305] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17169.867816] Lustre: lustre-MDT0000: new disk, initializing [17170.000916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17170.059195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17176.859827] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [17190.167979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17190.340021] Lustre: 416993:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17190.402102] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17190.408677] Lustre: Skipped 1 previous similar message [17190.480406] Lustre: lustre-MDT0001: new disk, initializing [17190.626878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17190.661849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17190.678707] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17196.023050] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [17206.333706] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [17211.955451] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [17221.337702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17221.671719] Lustre: lustre-OST0000: new disk, initializing [17221.676187] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17221.684213] Lustre: 418734:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17221.785518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17228.370391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17228.381331] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17228.580157] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17230.959717] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [17240.838927] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [17252.328505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17252.346620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17252.369474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17254.370369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17254.377051] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17254.433088] Lustre: Skipped 1 previous similar message [17257.400707] Lustre: server umount lustre-MDT0000 complete [17262.935761] LustreError: 416982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779710248 with bad export cookie 9634976304634898138 [17262.951248] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17262.954739] LustreError: 416982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [17263.458688] Lustre: server umount lustre-MDT0001 complete [17279.251401] Lustre: server umount lustre-OST0000 complete [17295.412386] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 07:57:57 (1779710277) [17306.458225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17306.984348] LustreError: 420654:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17307.007876] LustreError: 420654:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [17307.080096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17312.228843] LustreError: 420655:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17312.250735] LustreError: 420655:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [17313.453882] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [17317.346869] LustreError: 420654:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17322.467754] LustreError: 420655:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17325.065205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17325.367455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17332.318483] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [17340.069795] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [17345.376518] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [17367.289187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17367.632976] Lustre: Found index 0 for lustre-OST0000, updating log [17367.983456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17375.888295] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [17378.313024] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [17384.163623] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [17393.640268] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17393.644883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17393.664766] Lustre: Skipped 1 previous similar message [17393.686029] Lustre: Skipped 3 previous similar messages [17397.867083] Lustre: server umount lustre-OST0000 complete [17416.170462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17416.184076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17416.201070] Lustre: Skipped 1 previous similar message [17416.231071] Lustre: Skipped 3 previous similar messages [17420.323746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17420.332803] Lustre: Skipped 1 previous similar message [17421.353807] LustreError: 421912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17421.640626] Lustre: server umount lustre-MDT0000 complete [17426.693953] LustreError: 420635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779710411 with bad export cookie 9634976304634899531 [17426.706434] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17426.711235] LustreError: 420635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [17426.979209] LNet: 1 local NIs in recovery (showing 1): 192.168.201.157@tcp [17427.159323] Lustre: server umount lustre-MDT0001 complete [17440.399569] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 08:00:22 (1779710422) [18090.947980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18091.757179] LustreError: 426744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18091.794495] LustreError: 426744:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [18091.947642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18097.132498] LustreError: 426745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18098.890547] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [18101.220171] LustreError: 426744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18107.370225] LustreError: 426745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18107.391761] LustreError: 426745:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [18111.940589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18112.496539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18120.005506] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [18129.796567] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [18135.925102] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [18143.905707] LNet: 1 local NIs in recovery (showing 1): 192.168.201.157@tcp [18160.169282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18160.587383] Lustre: Found index 0 for lustre-OST0000, updating log [18161.051942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18165.196279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [18169.782379] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [18178.929894] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [18185.329827] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18185.369402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18188.686592] Lustre: server umount lustre-OST0000 complete [18196.259042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18196.262170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18196.288175] Lustre: Skipped 1 previous similar message [18196.321627] Lustre: Skipped 1 previous similar message [18198.370705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18198.377266] Lustre: Skipped 3 previous similar messages [18201.813780] Lustre: server umount lustre-MDT0000 complete [18202.470048] LustreError: 426744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18202.492985] LustreError: 426744:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [18207.189849] LustreError: 428369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779711192 with bad export cookie 9634976304634900616 [18207.198585] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18207.214038] LustreError: 428369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [18207.669443] Lustre: server umount lustre-MDT0001 complete [18210.531135] LNet: 1 local NIs in recovery (showing 1): 192.168.201.157@tcp [18249.235838] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [18268.673869] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [18323.262310] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 08:15:04 (1779711304) [18351.579697] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [18355.909637] Key type lgssc unregistered [18356.521780] LNet: 433012:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18356.546796] LNetError: 433012:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [18486.633927] LNet: Removed LNI 192.168.201.157@tcp [18487.920206] Key type .llcrypt unregistered [18487.925719] Key type ._llcrypt unregistered [18513.314492] Key type ._llcrypt registered [18513.317778] Key type .llcrypt registered [18513.504270] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [18515.395094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [18515.458839] alg: No test for adler32 (adler32-zlib) [18516.601466] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [18516.822991] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [18518.520370] Key type lgssc registered [18520.063541] Lustre: Echo OBD driver; http://www.lustre.org/ [18526.167773] LNet: Added LNI 192.168.201.157@tcp100 [8/256/0/180] [18527.486858] LNet: Added LNI 192.168.201.157@tcp101 [8/256/0/180] [18528.571690] LNet: Added LNI 192.168.201.157@tcp102 [8/256/0/180] [18529.510726] LNet: Added LNI 192.168.201.157@tcp103 [8/256/0/180] [18530.810206] LNet: Added LNI 192.168.201.157@tcp104 [8/256/0/180] [18532.181725] LNet: Added LNI 192.168.201.157@tcp105 [8/256/0/180] [18533.450513] LNet: Added LNI 192.168.201.157@tcp106 [8/256/0/180] [18534.802574] LNet: Added LNI 192.168.201.157@tcp107 [8/256/0/180] [18535.991480] LNet: Added LNI 192.168.201.157@tcp108 [8/256/0/180] [18537.511703] LNet: Added LNI 192.168.201.157@tcp109 [8/256/0/180] [18538.526738] LNet: Added LNI 192.168.201.157@tcp110 [8/256/0/180] [18539.560507] LNet: Added LNI 192.168.201.157@tcp111 [8/256/0/180] [18540.674458] LNet: Added LNI 192.168.201.157@tcp112 [8/256/0/180] [18541.839853] LNet: Added LNI 192.168.201.157@tcp113 [8/256/0/180] [18543.117372] LNet: Added LNI 192.168.201.157@tcp114 [8/256/0/180] [18544.436342] LNet: Added LNI 192.168.201.157@tcp115 [8/256/0/180] [18545.577697] LNet: Added LNI 192.168.201.157@tcp116 [8/256/0/180] [18546.795736] LNet: Added LNI 192.168.201.157@tcp117 [8/256/0/180] [18547.863684] LNet: Added LNI 192.168.201.157@tcp118 [8/256/0/180] [18548.966184] LNet: Added LNI 192.168.201.157@tcp119 [8/256/0/180] [18550.081066] LNet: Added LNI 192.168.201.157@tcp120 [8/256/0/180] [18551.370912] LNet: Added LNI 192.168.201.157@tcp121 [8/256/0/180] [18552.433383] LNet: Added LNI 192.168.201.157@tcp122 [8/256/0/180] [18554.066216] LNet: Added LNI 192.168.201.157@tcp123 [8/256/0/180] [18555.519708] LNet: Added LNI 192.168.201.157@tcp124 [8/256/0/180] [18556.934365] LNet: Added LNI 192.168.201.157@tcp125 [8/256/0/180] [18558.274606] LNet: Added LNI 192.168.201.157@tcp126 [8/256/0/180] [18559.812746] LNet: Added LNI 192.168.201.157@tcp127 [8/256/0/180] [18561.163916] LNet: Added LNI 192.168.201.157@tcp128 [8/256/0/180] [18562.438549] LNet: Added LNI 192.168.201.157@tcp129 [8/256/0/180] [18563.562869] LNet: Added LNI 192.168.201.157@tcp130 [8/256/0/180] [18564.707429] LNet: Added LNI 192.168.201.157@tcp131 [8/256/0/180] [18565.885698] LNet: Added LNI 192.168.201.157@tcp132 [8/256/0/180] [18567.232125] LNet: Added LNI 192.168.201.157@tcp133 [8/256/0/180] [18568.440626] LNet: Added LNI 192.168.201.157@tcp134 [8/256/0/180] [18569.592227] LNet: Added LNI 192.168.201.157@tcp135 [8/256/0/180] [18579.286706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [18579.323638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18580.511437] Lustre: 436034:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.157@tcp: can't add NID 192.168.201.157@tcp130: rc = -75 [18580.695373] LNetError: 433492:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.157 [18580.719326] LNetError: Protocol error connecting to 192.168.201.157@tcp115 at host 192.168.201.157:988: is it running a compatible version of Lustre? [18580.731733] LNetError: Skipped 1 previous similar message [18580.914607] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [18580.989964] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [18581.109123] Lustre: lustre-MDT0000: new disk, initializing [18581.212126] LNetError: 436054:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.157 [18581.239427] LNetError: 436054:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [18581.247188] LNetError: Protocol error connecting to 192.168.201.157@tcp102 at host 192.168.201.157:988: is it running a compatible version of Lustre? [18581.248909] Lustre: 436093:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.157@tcp: can't add NID 192.168.201.157@tcp130: rc = -75 [18581.264548] LNetError: Skipped 7 previous similar messages [18581.282288] Lustre: 436093:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [18581.305625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18581.346296] LNet: 4 local NIs in recovery (showing 4): 192.168.201.157@tcp128, 192.168.201.157@tcp129, 192.168.201.157@tcp126, 192.168.201.157@tcp130 [18581.368229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [18582.440625] LNetError: 436054:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.157 [18582.463072] LNetError: 436054:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [18582.476481] LNetError: Protocol error connecting to 192.168.201.157@tcp130 at host 192.168.201.157:988: is it running a compatible version of Lustre? [18582.508255] LNetError: Skipped 10 previous similar messages [18584.495385] LNetError: 436067:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.157 [18584.514405] LNetError: 436067:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [18584.527394] LNetError: Protocol error connecting to 192.168.201.157@tcp130 at host 192.168.201.157:988: is it running a compatible version of Lustre? [18584.548289] LNetError: Skipped 2 previous similar messages [18586.804596] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [18588.594937] LNetError: 433492:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.157 [18588.628215] LNetError: 433492:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [18588.652226] LNetError: Protocol error connecting to 192.168.201.157@tcp130 at host 192.168.201.157:988: is it running a compatible version of Lustre? [18588.673978] LNetError: Skipped 2 previous similar messages [18596.769903] LNetError: 436069:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.157 [18596.795224] LNetError: 436069:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [18596.816718] LNetError: Protocol error connecting to 192.168.201.157@tcp130 at host 192.168.201.157:988: is it running a compatible version of Lustre? [18596.836657] LNetError: Skipped 2 previous similar messages [18600.312749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18600.524584] Lustre: 436050:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [18600.618633] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [18600.627702] Lustre: Skipped 1 previous similar message [18600.820157] Lustre: lustre-MDT0001: new disk, initializing [18600.911346] Lustre: 436837:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.157@tcp: can't add NID 192.168.201.157@tcp130: rc = -75 [18600.941226] Lustre: 436837:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [18601.172449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18601.216869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [18601.243736] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [18606.181188] Lustre: 437043:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.157@tcp: can't add NID 192.168.201.157@tcp130: rc = -75 [18606.203630] Lustre: 437043:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [18606.487179] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [18613.095941] LNetError: 436060:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.157 [18613.126776] LNetError: 436060:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 27 previous similar messages [18613.153200] LNetError: Protocol error connecting to 192.168.201.157@tcp107 at host 192.168.201.157:988: is it running a compatible version of Lustre? [18613.180529] LNetError: Skipped 27 previous similar messages [18616.544556] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [18623.808336] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [18633.231708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18633.570061] Lustre: lustre-OST0000: new disk, initializing [18633.578901] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [18633.594442] Lustre: 437803:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [18633.768993] Lustre: 437808:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.157@tcp: can't add NID 192.168.201.157@tcp130: rc = -75 [18633.780107] Lustre: 437808:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [18633.848317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18641.040584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [18641.052507] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [18641.246059] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [18641.701282] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [18641.835246] LNet: 5 local NIs in recovery (showing 5): 192.168.201.157@tcp135, 192.168.201.157@tcp131, 192.168.201.157@tcp130, 192.168.201.157@tcp128, 192.168.201.157@tcp129 [18645.857942] LNetError: 436062:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.157 [18645.880573] LNetError: 436062:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 37 previous similar messages [18645.892391] LNetError: Protocol error connecting to 192.168.201.157@tcp124 at host 192.168.201.157:988: is it running a compatible version of Lustre? [18645.917980] LNetError: Skipped 37 previous similar messages [18652.178849] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [18666.978940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [18666.981176] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18666.996527] LustreError: Skipped 1 previous similar message [18666.999545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18667.032099] Lustre: Skipped 1 previous similar message [18667.060175] Lustre: Skipped 1 previous similar message [18668.437567] Lustre: server umount lustre-OST0000 complete [18675.178224] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18675.185301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18675.199821] Lustre: Skipped 1 previous similar message [18680.507580] Lustre: server umount lustre-MDT0000 complete [18685.423556] LustreError: 436059:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18685.472064] LustreError: 436059:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [18685.810830] LustreError: 436042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779711670 with bad export cookie 4234767620063117601 [18685.818561] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18685.833084] LustreError: 436042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [18686.243426] Lustre: server umount lustre-MDT0001 complete [18704.927111] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [18708.292139] Key type lgssc unregistered [18709.052594] LNet: 439609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18709.068165] LNetError: 439609:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [18709.088283] LNet: Removed LNI 192.168.201.157@tcp [18710.112178] LNet: Removed LNI 192.168.201.157@tcp100 [18710.114603] LNetError: 439609:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [18710.134584] LNetError: 439609:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [18710.146931] LNet: Removed LNI 192.168.201.157@tcp101 [18711.200181] LNet: Removed LNI 192.168.201.157@tcp102 [18711.223421] LNetError: 439609:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [18711.259581] LNetError: 439609:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [18711.290448] LNet: Removed LNI 192.168.201.157@tcp103 [18712.352638] LNet: Removed LNI 192.168.201.157@tcp104 [18713.440250] LNet: Removed LNI 192.168.201.157@tcp105 [18713.444217] LNetError: 439609:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [18713.476107] LNetError: 439609:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [18713.503534] LNet: Removed LNI 192.168.201.157@tcp106 [18714.532635] LNet: Removed LNI 192.168.201.157@tcp107 [18715.552122] LNet: Removed LNI 192.168.201.157@tcp108 [18716.578877] LNet: Removed LNI 192.168.201.157@tcp109 [18717.600653] LNet: Removed LNI 192.168.201.157@tcp110 [18717.616110] LNetError: 439609:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [18717.630031] LNetError: 439609:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [18717.653727] LNet: Removed LNI 192.168.201.157@tcp111 [18718.688307] LNet: Removed LNI 192.168.201.157@tcp112 [18719.715087] LNet: Removed LNI 192.168.201.157@tcp113 [18720.736560] LNet: Removed LNI 192.168.201.157@tcp114 [18721.762487] LNet: Removed LNI 192.168.201.157@tcp115 [18722.787040] LNet: Removed LNI 192.168.201.157@tcp116 [18723.808994] LNet: Removed LNI 192.168.201.157@tcp117 [18724.832884] LNet: Removed LNI 192.168.201.157@tcp118 [18725.857297] LNet: Removed LNI 192.168.201.157@tcp119 [18725.861160] LNetError: 439609:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [18725.891719] LNetError: 439609:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [18725.898905] LNet: Removed LNI 192.168.201.157@tcp120 [18726.944416] LNet: Removed LNI 192.168.201.157@tcp121 [18728.032184] LNet: Removed LNI 192.168.201.157@tcp122 [18729.060804] LNet: Removed LNI 192.168.201.157@tcp123 [18730.080340] LNet: Removed LNI 192.168.201.157@tcp124 [18731.104320] LNet: Removed LNI 192.168.201.157@tcp125 [18732.128397] LNet: Removed LNI 192.168.201.157@tcp126 [18733.216659] LNet: Removed LNI 192.168.201.157@tcp127 [18733.221900] LNet: Removed LNI 192.168.201.157@tcp128 [18733.230981] LNet: Removed LNI 192.168.201.157@tcp129 [18733.239657] LNet: Removed LNI 192.168.201.157@tcp130 [18733.248418] LNet: Removed LNI 192.168.201.157@tcp131 [18734.305368] LNet: Removed LNI 192.168.201.157@tcp132 [18735.333294] LNet: Removed LNI 192.168.201.157@tcp133 [18736.419347] LNet: Removed LNI 192.168.201.157@tcp134 [18736.433060] LNet: Removed LNI 192.168.201.157@tcp135 [18737.707708] Key type .llcrypt unregistered [18737.709746] Key type ._llcrypt unregistered [18772.266749] Key type ._llcrypt registered [18772.270176] Key type .llcrypt registered [18772.384506] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [18794.158751] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [18796.118194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [18796.242143] alg: No test for adler32 (adler32-zlib) [18797.390292] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [18797.742290] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [18799.481935] Key type lgssc registered [18801.040564] Lustre: Echo OBD driver; http://www.lustre.org/ [18857.622434] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 08:23:59 (1779711839) [18888.794408] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [18893.022529] Key type lgssc unregistered [18893.476785] LNet: 443657:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18893.489126] LNetError: 443657:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [18893.523808] LNet: Removed LNI 192.168.201.157@tcp [18895.034597] Key type .llcrypt unregistered [18895.038570] Key type ._llcrypt unregistered [18920.744684] Key type ._llcrypt registered [18920.747166] Key type .llcrypt registered [18920.907770] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [18923.018961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [18923.075198] alg: No test for adler32 (adler32-zlib) [18924.289977] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [18924.701201] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [18926.576534] Key type lgssc registered [18928.129853] Lustre: Echo OBD driver; http://www.lustre.org/ [18940.724842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [18940.752566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18942.257388] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [18942.328345] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [18942.458603] Lustre: lustre-MDT0000: new disk, initializing [18942.542831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18942.564681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [18948.386327] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [18962.756829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18963.025622] Lustre: 444861:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [18963.140722] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [18963.148833] Lustre: Skipped 1 previous similar message [18963.306169] Lustre: lustre-MDT0001: new disk, initializing [18963.497216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18963.558444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [18963.580935] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [18969.673816] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [18980.878373] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [18986.498709] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [18997.031085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18997.455547] Lustre: lustre-OST0000: new disk, initializing [18997.468251] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [18997.483720] Lustre: 446605:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [18997.613818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19003.998575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [19004.008154] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [19004.152948] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [19005.043163] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [19015.150316] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [19019.699979] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing lnet_if_list [19026.801454] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [19026.871690] LNet: Added LNI 192.168.201.157@tcp10 [8/256/0/180] [19034.310182] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [19034.344562] LNet: Added LNI 192.168.201.157@tcp11 [8/256/0/180] [19039.963573] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [19040.005946] LNet: Added LNI 192.168.201.157@tcp12 [8/256/0/180] [19045.873513] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [19045.910565] LNet: Added LNI 192.168.201.157@tcp13 [8/256/0/180] [19051.944090] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [19051.983364] LNet: Added LNI 192.168.201.157@tcp14 [8/256/0/180] [19058.677065] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [19058.741299] LNet: Added LNI 192.168.201.157@tcp15 [8/256/0/180] [19065.303433] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [19065.339382] LNet: Added LNI 192.168.201.157@tcp16 [8/256/0/180] [19071.616564] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [19071.652332] LNet: Added LNI 192.168.201.157@tcp17 [8/256/0/180] [19078.291984] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [19078.344624] LNet: Added LNI 192.168.201.157@tcp18 [8/256/0/180] [19085.821618] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [19085.883146] LNet: Added LNI 192.168.201.157@tcp19 [8/256/0/180] [19093.385705] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [19093.428042] LNet: Added LNI 192.168.201.157@tcp20 [8/256/0/180] [19101.126515] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [19101.178968] LNet: Added LNI 192.168.201.157@tcp21 [8/256/0/180] [19109.343577] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [19109.397147] LNet: Added LNI 192.168.201.157@tcp22 [8/256/0/180] [19116.164308] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [19116.191853] LNet: Added LNI 192.168.201.157@tcp23 [8/256/0/180] [19123.540764] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [19123.592906] LNet: Added LNI 192.168.201.157@tcp24 [8/256/0/180] [19130.502271] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [19130.544963] LNet: Added LNI 192.168.201.157@tcp25 [8/256/0/180] [19139.038221] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [19139.079848] LNet: Added LNI 192.168.201.157@tcp26 [8/256/0/180] [19147.658096] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [19147.711241] LNet: Added LNI 192.168.201.157@tcp27 [8/256/0/180] [19156.165125] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [19156.220475] LNet: Added LNI 192.168.201.157@tcp28 [8/256/0/180] [19165.310574] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [19165.354795] LNet: Added LNI 192.168.201.157@tcp29 [8/256/0/180] [19173.256623] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [19173.278107] LNet: Added LNI 192.168.201.157@tcp30 [8/256/0/180] [19182.315900] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [19182.349637] LNet: Added LNI 192.168.201.157@tcp31 [8/256/0/180] [19191.002209] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [19191.044119] LNet: Added LNI 192.168.201.157@tcp32 [8/256/0/180] [19200.460914] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [19200.568976] LNet: Added LNI 192.168.201.157@tcp33 [8/256/0/180] [19210.192981] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [19210.227312] LNet: Added LNI 192.168.201.157@tcp34 [8/256/0/180] [19219.759981] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [19219.829664] LNet: Added LNI 192.168.201.157@tcp35 [8/256/0/180] [19230.700254] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [19230.729284] LNet: Added LNI 192.168.201.157@tcp36 [8/256/0/180] [19239.869466] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [19239.912635] LNet: Added LNI 192.168.201.157@tcp37 [8/256/0/180] [19251.639314] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [19251.694916] LNet: Added LNI 192.168.201.157@tcp38 [8/256/0/180] [19262.557409] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [19262.603549] LNet: Added LNI 192.168.201.157@tcp39 [8/256/0/180] [19271.437398] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [19271.504378] LNet: Added LNI 192.168.201.157@tcp40 [8/256/0/180] [19281.890123] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [19281.938968] LNet: Added LNI 192.168.201.157@tcp41 [8/256/0/180] [19291.787813] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [19291.834578] LNet: Added LNI 192.168.201.157@tcp42 [8/256/0/180] [19301.861785] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [19301.926739] LNet: Added LNI 192.168.201.157@tcp43 [8/256/0/180] [19312.483305] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [19312.548301] LNet: Added LNI 192.168.201.157@tcp44 [8/256/0/180] [19323.140989] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [19323.189055] LNet: Added LNI 192.168.201.157@tcp45 [8/256/0/180] [19334.970126] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [19335.053779] LNet: Added LNI 192.168.201.157@tcp46 [8/256/0/180] [19346.430971] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [19346.450319] LNet: Added LNI 192.168.201.157@tcp47 [8/256/0/180] [19356.732271] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [19356.777674] LNet: Added LNI 192.168.201.157@tcp48 [8/256/0/180] [19368.413496] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [19368.449637] LNet: Added LNI 192.168.201.157@tcp49 [8/256/0/180] [19380.523469] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [19380.558046] LNet: Added LNI 192.168.201.157@tcp50 [8/256/0/180] [19408.877571] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19408.889992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19408.910173] Lustre: Skipped 1 previous similar message [19408.928294] Lustre: Skipped 2 previous similar messages [19414.057847] Lustre: server umount lustre-MDT0000 complete [19419.114719] LustreError: 446845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19419.157622] LustreError: 446845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [19420.154775] LustreError: 446604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779712405 with bad export cookie 15010957945606815940 [19420.172972] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19420.174927] LustreError: 446604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [19420.953911] Lustre: server umount lustre-MDT0001 complete [19440.609984] Lustre: 444132:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779712409/real 1779712409] req@0000000026d5bcda x1866163194894848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779712425 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19440.647607] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19440.680237] Lustre: Skipped 1 previous similar message [19441.633249] Lustre: 444133:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779712410/real 1779712410] req@00000000841133b4 x1866163194894976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779712426 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19445.796367] Lustre: 444133:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779712414/real 1779712414] req@0000000012e01c28 x1866163194895104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779712430 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19448.683234] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [19451.881490] Lustre: 444132:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779712420/real 1779712420] req@0000000026b2cb5e x1866163194895488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779712436 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19451.921346] Lustre: 444132:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [19455.968545] Lustre: 444134:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779712425/real 1779712425] req@0000000017259fd8 x1866163194895616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779712441 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19461.140458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19466.346957] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd051a2b85638cb21 [19466.375248] Lustre: MGC192.168.201.157@tcp: Connection restored to 0@lo (at 0@lo) [19466.709230] LustreError: 446597:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19466.803810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19470.824882] LustreError: 446598:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19473.303642] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [19473.894671] LustreError: 446597:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19473.906865] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [19479.014245] LustreError: 446598:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19484.220646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19484.648244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19485.677368] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [19489.828178] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [19496.823314] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [19503.018697] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [19533.282387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [19533.302476] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19533.337036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19535.842447] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [19535.859969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19535.893150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19539.549691] Lustre: server umount lustre-OST0000 complete [19547.114813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19547.118234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19547.155572] Lustre: Skipped 1 previous similar message [19547.186457] Lustre: Skipped 3 previous similar messages [19552.237511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19552.261711] Lustre: Skipped 1 previous similar message [19553.156646] Lustre: server umount lustre-MDT0000 complete [19557.347099] LustreError: 470034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19557.380308] LustreError: 470034:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [19559.431333] LustreError: 444853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779712544 with bad export cookie 15010957945606818593 [19559.448196] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19559.466934] LustreError: 444853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [19560.003801] Lustre: server umount lustre-MDT0001 complete [19578.147431] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [19583.146370] Key type lgssc unregistered [19584.002061] LNet: 472601:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19584.015582] LNetError: 472601:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [19584.026389] LNet: Removed LNI 192.168.201.157@tcp [19584.047548] LNet: Removed LNI 192.168.201.157@tcp10 [19584.055830] LNet: Removed LNI 192.168.201.157@tcp11 [19584.068714] LNet: Removed LNI 192.168.201.157@tcp12 [19584.077833] LNet: Removed LNI 192.168.201.157@tcp13 [19584.079957] LNet: Removed LNI 192.168.201.157@tcp14 [19584.086448] LNet: Removed LNI 192.168.201.157@tcp15 [19584.095584] LNet: Removed LNI 192.168.201.157@tcp16 [19584.105693] LNet: Removed LNI 192.168.201.157@tcp17 [19584.114842] LNet: Removed LNI 192.168.201.157@tcp18 [19584.137056] LNet: Removed LNI 192.168.201.157@tcp19 [19584.148171] LNet: Removed LNI 192.168.201.157@tcp20 [19584.150246] LNet: Removed LNI 192.168.201.157@tcp21 [19584.155512] LNet: Removed LNI 192.168.201.157@tcp22 [19584.162334] LNet: Removed LNI 192.168.201.157@tcp23 [19584.170155] LNet: Removed LNI 192.168.201.157@tcp24 [19584.191035] LNet: Removed LNI 192.168.201.157@tcp25 [19584.197510] LNet: Removed LNI 192.168.201.157@tcp26 [19584.200378] LNet: Removed LNI 192.168.201.157@tcp27 [19584.210445] LNet: Removed LNI 192.168.201.157@tcp28 [19584.215522] LNet: Removed LNI 192.168.201.157@tcp29 [19584.232097] LNet: Removed LNI 192.168.201.157@tcp30 [19584.239735] LNet: Removed LNI 192.168.201.157@tcp31 [19584.242186] LNet: Removed LNI 192.168.201.157@tcp32 [19584.256103] LNet: Removed LNI 192.168.201.157@tcp33 [19584.272236] LNet: Removed LNI 192.168.201.157@tcp34 [19584.274293] LNet: Removed LNI 192.168.201.157@tcp35 [19584.280345] LNet: Removed LNI 192.168.201.157@tcp36 [19584.295550] LNet: Removed LNI 192.168.201.157@tcp37 [19584.304298] LNet: Removed LNI 192.168.201.157@tcp38 [19584.318355] LNet: Removed LNI 192.168.201.157@tcp39 [19584.332360] LNet: Removed LNI 192.168.201.157@tcp40 [19584.346160] LNet: Removed LNI 192.168.201.157@tcp41 [19584.359574] LNet: Removed LNI 192.168.201.157@tcp42 [19584.361822] LNet: Removed LNI 192.168.201.157@tcp43 [19584.366915] LNet: Removed LNI 192.168.201.157@tcp44 [19584.377189] LNet: Removed LNI 192.168.201.157@tcp45 [19584.379728] LNet: Removed LNI 192.168.201.157@tcp46 [19584.391374] LNet: Removed LNI 192.168.201.157@tcp47 [19584.393521] LNet: Removed LNI 192.168.201.157@tcp48 [19584.397166] LNet: Removed LNI 192.168.201.157@tcp49 [19584.409963] LNet: Removed LNI 192.168.201.157@tcp50 [19585.582985] Key type .llcrypt unregistered [19585.585794] Key type ._llcrypt unregistered [19621.566761] Key type ._llcrypt registered [19621.575287] Key type .llcrypt registered [19621.809153] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [19643.636477] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [19645.968703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19646.014049] alg: No test for adler32 (adler32-zlib) [19647.387694] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [19647.737752] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [19649.531336] Key type lgssc registered [19650.988482] Lustre: Echo OBD driver; http://www.lustre.org/ [19706.406855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [19706.428657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19707.922180] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [19707.977950] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [19708.087539] Lustre: lustre-MDT0000: new disk, initializing [19708.245933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19708.278840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [19715.272964] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [19730.585927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19730.683885] Lustre: 475773:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [19730.809977] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [19730.818270] Lustre: Skipped 1 previous similar message [19730.941257] Lustre: lustre-MDT0001: new disk, initializing [19731.053407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19731.093733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [19731.115968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [19737.495649] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [19749.042162] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [19755.150329] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [19766.885580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19767.205937] Lustre: *** cfs_fail_loc=90f, val=0*** [19767.402620] Lustre: lustre-OST0000: new disk, initializing [19767.415725] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [19767.447928] Lustre: 477564:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [19767.688395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19776.434410] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [19777.140418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [19777.155846] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [19777.346793] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [19786.360225] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [19797.986951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19797.992311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19798.015038] Lustre: Skipped 1 previous similar message [19803.106896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19803.124813] Lustre: Skipped 2 previous similar messages [19808.235238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19808.239565] Lustre: Skipped 1 previous similar message [19811.297313] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [19811.588134] Lustre: server umount lustre-OST0000 complete [19818.466279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19818.468355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19818.505271] Lustre: Skipped 1 previous similar message [19818.552310] Lustre: Skipped 1 previous similar message [19823.585923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19823.611407] Lustre: Skipped 1 previous similar message [19824.583360] Lustre: server umount lustre-MDT0000 complete [19831.362487] LustreError: 475764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779712816 with bad export cookie 5836316622234061629 [19831.363307] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19831.370980] LustreError: 475764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [19831.960763] Lustre: server umount lustre-MDT0001 complete [19850.733287] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [19854.743298] Key type lgssc unregistered [19855.072783] LNet: 479369:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19855.083706] LNetError: 479369:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [19855.126454] LNet: Removed LNI 192.168.201.157@tcp [19856.200919] Key type .llcrypt unregistered [19856.203578] Key type ._llcrypt unregistered [19888.296317] Key type ._llcrypt registered [19888.298707] Key type .llcrypt registered [19888.533451] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [19890.657446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19890.967737] alg: No test for adler32 (adler32-zlib) [19892.486985] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [19892.970790] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [19894.752551] Key type lgssc registered [19897.100067] Lustre: Echo OBD driver; http://www.lustre.org/ [19910.560645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [19910.587556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19912.152637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19919.297532] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [19922.403745] LustreError: 480625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19922.444788] LustreError: 480625:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [19927.523498] LustreError: 480626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19932.641585] LustreError: 480625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19932.733800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19933.462770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19939.701806] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [19947.679517] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [19952.724849] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [19963.997478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19964.549210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19972.524570] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [19980.400978] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [19998.085884] loop0: detected capacity change from 0 to 204800000 [19998.336269] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [19998.377163] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [19998.393355] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20007.518937] loop0: detected capacity change from 0 to 204800000 [20007.779226] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20007.801358] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20007.829139] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20014.866477] loop0: detected capacity change from 0 to 204800000 [20023.559660] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20023.694204] Lustre: fs15246-OST0000: new disk, initializing [20023.699817] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [20023.706446] Lustre: 483885:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [20023.807973] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [20032.762986] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [20046.437416] loop1: detected capacity change from 0 to 204800000 [20054.751790] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20054.950806] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [20055.046459] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [20055.147629] Lustre: fs15246-MDT0000: new disk, initializing [20055.446587] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20055.476331] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [20055.581790] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [20056.637657] Lustre: 484992:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000cbf49113 x1866164210818816/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1779713057 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [20060.516926] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [20065.777116] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [20065.794735] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [20065.862682] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [20095.984622] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20096.021064] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [20097.945194] Lustre: server umount fs15246-MDT0000 complete [20101.097163] LustreError: 480622:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20106.220232] LustreError: 483853:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20110.988957] Lustre: server umount fs15246-OST0000 complete [20126.694348] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [20126.703917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20126.707093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20126.736462] Lustre: Skipped 1 previous similar message [20131.823611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20131.834974] Lustre: Skipped 1 previous similar message [20132.658829] Lustre: server umount lustre-OST0000 complete [20138.469185] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [20138.482894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20138.523391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20138.531755] Lustre: Skipped 1 previous similar message [20142.053303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20144.991973] Lustre: server umount lustre-MDT0000 complete [20147.170262] LustreError: 480621:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20147.188259] LustreError: 480621:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [20150.031528] LustreError: 480606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779713135 with bad export cookie 15810798961159471030 [20150.037518] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20150.043041] LustreError: 480606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [20150.462947] Lustre: server umount lustre-MDT0001 complete [20166.271180] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [20170.289784] Key type lgssc unregistered [20170.589146] LNet: 488058:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [20170.597637] LNetError: 488058:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [20170.617708] LNet: Removed LNI 192.168.201.157@tcp [20171.679337] Key type .llcrypt unregistered [20171.681855] Key type ._llcrypt unregistered [20195.897750] Key type ._llcrypt registered [20195.900975] Key type .llcrypt registered [20196.087266] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [20197.490750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [20197.535909] alg: No test for adler32 (adler32-zlib) [20198.940710] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [20199.221048] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [20200.952372] Key type lgssc registered [20202.574362] Lustre: Echo OBD driver; http://www.lustre.org/ [20263.255648] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [20277.786718] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [20328.707615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [20328.775932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20330.252091] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [20330.290837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [20330.456500] Lustre: lustre-MDT0000: new disk, initializing [20330.577952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20330.598158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [20336.223189] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [20351.300959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20351.465358] Lustre: 492499:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [20351.540673] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [20351.544352] Lustre: Skipped 1 previous similar message [20351.693461] Lustre: lustre-MDT0001: new disk, initializing [20351.829905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [20351.899200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [20351.927910] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [20357.673854] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [20367.532289] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [20373.362901] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [20383.501803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20384.083122] Lustre: lustre-OST0000: new disk, initializing [20384.088455] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [20384.098104] Lustre: 494241:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [20384.266106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [20392.620195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [20392.647388] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [20392.827600] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [20392.906328] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [20402.495576] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [20408.340681] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [20408.556682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [20413.636972] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [20413.919983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [20418.530742] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [20418.534675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20418.541655] LustreError: Skipped 1 previous similar message [20418.545485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20418.603415] Lustre: Skipped 1 previous similar message [20422.620385] Lustre: server umount lustre-OST0000 complete [20428.776878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20428.787618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [20428.794138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20428.794147] Lustre: Skipped 1 previous similar message [20428.798988] Lustre: Skipped 1 previous similar message [20433.894647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20433.907481] Lustre: Skipped 1 previous similar message [20434.130696] Lustre: server umount lustre-MDT0000 complete [20439.660470] LustreError: 492491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779713424 with bad export cookie 10249447000651717883 [20439.682733] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20439.686843] LustreError: 492491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [20440.258706] Lustre: server umount lustre-MDT0001 complete [20477.585957] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [20493.772962] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [20540.270710] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 20120 sec ========= 08:52:02 (1779713522) [20542.689203] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:52:04 (1779713524) === [20548.382594] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:52:09 (1779713529) === [20611.053285] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [20615.506678] Key type lgssc unregistered [20616.083443] LNet: 500735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [20616.115024] LNetError: 500735:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [20616.181054] LNet: Removed LNI 192.168.201.157@tcp [20617.776685] Key type .llcrypt unregistered [20617.780805] Key type ._llcrypt unregistered