[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 513654825 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002394] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.006012] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.012040] LSM: Security Framework initializing [ 0.013051] Yama: becoming mindful. [ 0.014036] SELinux: Initializing. [ 0.015091] *** VALIDATE selinux *** [ 0.024267] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029020] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030161] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031143] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032137] *** VALIDATE tmpfs *** [ 0.034459] *** VALIDATE proc *** [ 0.036068] *** VALIDATE cgroup *** [ 0.037009] *** VALIDATE cgroup2 *** [ 0.039212] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040175] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041015] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042037] Spectre V2 : User space: Vulnerable [ 0.044007] Speculative Store Bypass: Vulnerable [ 0.046853] debug: unmapping init [mem 0xffffffffa0e59000-0xffffffffa0e60fff] [ 0.048829] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049777] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050032] ... version: 2 [ 0.051020] ... bit width: 48 [ 0.052015] ... generic registers: 4 [ 0.053014] ... value mask: 0000ffffffffffff [ 0.054022] ... max period: 00007fffffffffff [ 0.055024] ... fixed-purpose events: 3 [ 0.056015] ... event mask: 000000070000000f [ 0.057352] rcu: Hierarchical SRCU implementation. [ 0.059613] smp: Bringing up secondary CPUs ... [ 0.060698] x86: Booting SMP configuration: [ 0.061029] .... node #0, CPUs: #1 #2 #3 [ 0.064517] smp: Brought up 1 node, 4 CPUs [ 0.066018] smpboot: Max logical packages: 1 [ 0.067031] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.135650] node 0 deferred pages initialised in 65ms [ 0.139132] devtmpfs: initialized [ 0.140341] x86/mm: Memory block size: 128MB [ 0.143952] gcov: version magic: 0x41383552 [ 0.147313] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.148108] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.149327] pinctrl core: initialized pinctrl subsystem [ 0.150213] [ 0.150778] ************************************************************* [ 0.151015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152019] ** ** [ 0.153019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.154019] ** ** [ 0.155016] ** This means that this kernel is built to expose internal ** [ 0.156022] ** IOMMU data structures, which may compromise security on ** [ 0.157024] ** your system. ** [ 0.158015] ** ** [ 0.159021] ** If you see this message and you are not debugging the ** [ 0.160019] ** kernel, report this immediately to your vendor! ** [ 0.161020] ** ** [ 0.162024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163018] ************************************************************* [ 0.164904] NET: Registered protocol family 16 [ 0.167579] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.170167] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.173102] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.177038] cpuidle: using governor menu [ 0.178940] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.181507] PCI: Using configuration type 1 for base access [ 0.183149] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.192063] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.194069] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.199035] cryptd: max_cpu_qlen set to 1000 [ 0.201698] ACPI: Added _OSI(Module Device) [ 0.202000] ACPI: Added _OSI(Processor Device) [ 0.204012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.206021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.211989] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.219151] ACPI: Interpreter enabled [ 0.221136] ACPI: PM: (supports S0 S3 S4 S5) [ 0.223031] ACPI: Using IOAPIC for interrupt routing [ 0.226308] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.231561] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.241789] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.244059] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.247023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.251092] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.257344] acpiphp: Slot [2] registered [ 0.259141] acpiphp: Slot [5] registered [ 0.260155] acpiphp: Slot [6] registered [ 0.262157] acpiphp: Slot [7] registered [ 0.263129] acpiphp: Slot [8] registered [ 0.265174] acpiphp: Slot [9] registered [ 0.267214] acpiphp: Slot [10] registered [ 0.269188] acpiphp: Slot [3] registered [ 0.270151] acpiphp: Slot [4] registered [ 0.272085] acpiphp: Slot [11] registered [ 0.273089] acpiphp: Slot [12] registered [ 0.275148] acpiphp: Slot [13] registered [ 0.276119] acpiphp: Slot [14] registered [ 0.278177] acpiphp: Slot [15] registered [ 0.279133] acpiphp: Slot [16] registered [ 0.281125] acpiphp: Slot [17] registered [ 0.283121] acpiphp: Slot [18] registered [ 0.285245] acpiphp: Slot [19] registered [ 0.287153] acpiphp: Slot [20] registered [ 0.289162] acpiphp: Slot [21] registered [ 0.291171] acpiphp: Slot [22] registered [ 0.293183] acpiphp: Slot [23] registered [ 0.295160] acpiphp: Slot [24] registered [ 0.297181] acpiphp: Slot [25] registered [ 0.300189] acpiphp: Slot [26] registered [ 0.302178] acpiphp: Slot [27] registered [ 0.305213] acpiphp: Slot [28] registered [ 0.307227] acpiphp: Slot [29] registered [ 0.309203] acpiphp: Slot [30] registered [ 0.312154] acpiphp: Slot [31] registered [ 0.316019] PCI host bridge to bus 0000:00 [ 0.318050] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.321045] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.323022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.325023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.327025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.330032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.332206] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.335082] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.338647] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.349765] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.354061] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.357024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.359021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.362029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.365054] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.367816] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.370120] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.373747] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.379017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.396034] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.403021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.408172] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.422020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.430025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.453026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.468132] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.477039] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.487026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.510018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.522196] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.531022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.538018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.565023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.576285] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.585025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.592023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.608021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.617591] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.624021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.631022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.660020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.668819] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.675016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.681019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.696032] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.712775] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.714437] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.717361] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.719490] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.721277] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.725061] iommu: Default domain type: Passthrough [ 0.727572] SCSI subsystem initialized [ 0.729173] ACPI: bus type USB registered [ 0.730175] usbcore: registered new interface driver usbfs [ 0.733115] usbcore: registered new interface driver hub [ 0.735130] usbcore: registered new device driver usb [ 0.736151] pps_core: LinuxPPS API ver. 1 registered [ 0.738011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.740085] PTP clock support registered [ 0.742164] EDAC MC: Ver: 3.0.0 [ 0.745184] PCI: Using ACPI for IRQ routing [ 0.746851] NetLabel: Initializing [ 0.748013] NetLabel: domain hash size = 128 [ 0.749015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.751090] NetLabel: unlabeled traffic allowed by default [ 0.753206] vgaarb: loaded [ 0.755306] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.757031] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.763359] clocksource: Switched to clocksource kvm-clock [ 0.940158] VFS: Disk quotas dquot_6.6.0 [ 0.943601] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.948899] *** VALIDATE ramfs *** [ 0.951480] *** VALIDATE hugetlbfs *** [ 0.954704] pnp: PnP ACPI init [ 0.960675] pnp: PnP ACPI: found 6 devices [ 1.029226] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.033362] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.036292] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.039028] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.041742] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.044879] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.047888] NET: Registered protocol family 2 [ 1.051061] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.057416] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.061685] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.068898] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.073250] TCP: Hash tables configured (established 65536 bind 65536) [ 1.076678] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.080665] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.083934] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.087313] NET: Registered protocol family 1 [ 1.090097] RPC: Registered named UNIX socket transport module. [ 1.092534] RPC: Registered udp transport module. [ 1.094124] RPC: Registered tcp transport module. [ 1.095782] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.098067] NET: Registered protocol family 44 [ 1.099952] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.102529] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.105084] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.107891] PCI: CLS 0 bytes, default 64 [ 1.109987] Unpacking initramfs... [ 2.580618] debug: unmapping init [mem 0xffff94b67cc54000-0xffff94b67ffbffff] [ 2.585097] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.587300] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.590043] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.121439] Initialise system trusted keyrings [ 3.124547] Key type blacklist registered [ 3.132196] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.143978] zbud: loaded [ 3.148100] *** VALIDATE nfs *** [ 3.149371] *** VALIDATE nfs4 *** [ 3.151175] pstore: using deflate compression [ 3.153935] Platform Keyring initialized [ 3.295737] NET: Registered protocol family 38 [ 3.297670] Key type asymmetric registered [ 3.299712] Asymmetric key parser 'x509' registered [ 3.302369] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.306291] io scheduler mq-deadline registered [ 3.308305] io scheduler kyber registered [ 3.310363] io scheduler bfq registered [ 3.312486] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.315812] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.319695] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.323523] ACPI: Power Button [PWRF] [ 3.332137] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.340952] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.362616] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.374522] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.395613] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.424223] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.458179] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.466181] Non-volatile memory driver v1.3 [ 3.468598] Linux agpgart interface v0.103 [ 3.514860] virtio_blk virtio1: [vda] 146864 512-byte logical blocks (75.2 MB/71.7 MiB) [ 3.520256] vda: detected capacity change from 0 to 75194368 [ 3.538932] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.542321] vdb: detected capacity change from 0 to 1073741824 [ 3.561918] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.565768] vdc: detected capacity change from 0 to 2621440000 [ 3.585645] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.589180] vdd: detected capacity change from 0 to 2621440000 [ 3.606082] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.609292] vde: detected capacity change from 0 to 4294967296 [ 3.624471] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.629159] vdf: detected capacity change from 0 to 4294967296 [ 3.652699] libphy: Fixed MDIO Bus: probed [ 3.659110] usbcore: registered new interface driver usbserial_generic [ 3.662169] usbserial: USB Serial support registered for generic [ 3.664955] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.670180] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.672537] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.675539] mousedev: PS/2 mouse device common for all mice [ 3.679129] rtc_cmos 00:05: RTC can wake from S4 [ 3.682251] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.687221] rtc_cmos 00:05: registered as rtc0 [ 3.693217] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.693510] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.696572] intel_pstate: CPU model not supported [ 3.703177] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.707814] hid: raw HID events driver (C) Jiri Kosina [ 3.708055] usbcore: registered new interface driver usbhid [ 3.708060] usbhid: USB HID core driver [ 3.708252] drop_monitor: Initializing network drop monitor service [ 3.708425] Initializing XFRM netlink socket [ 3.708826] NET: Registered protocol family 10 [ 3.710583] Segment Routing with IPv6 [ 3.725775] NET: Registered protocol family 17 [ 3.728417] mpls_gso: MPLS GSO support [ 3.735410] RAS: Correctable Errors collector initialized. [ 3.738305] AVX version of gcm_enc/dec engaged. [ 3.740354] AES CTR mode by8 optimization enabled [ 3.855892] sched_clock: Marking stable (3855866603, 0)->(4814694745, -958828142) [ 3.862389] registered taskstats version 1 [ 3.866263] Loading compiled-in X.509 certificates [ 3.868238] zswap: loaded using pool lzo/zbud [ 3.905377] Key type big_key registered [ 3.919678] Key type encrypted registered [ 3.921611] ima: No TPM chip found, activating TPM-bypass! [ 3.924696] ima: Allocated hash algorithm: sha1 [ 3.926848] ima: No architecture policies found [ 3.929306] evm: Initialising EVM extended attributes: [ 3.931779] evm: security.selinux [ 3.933493] evm: security.ima [ 3.934636] evm: security.capability [ 3.935944] evm: HMAC attrs: 0x1 [ 3.939166] rtc_cmos 00:05: setting system clock to 2026-08-27 17:48:42 UTC (1787852922) [ 3.945393] debug: unmapping init [mem 0xffffffffa1e03000-0xffffffffa1ffffff] [ 3.948386] debug: unmapping init [mem 0xffffffffa0b82000-0xffffffffa0e58fff] [ 3.959545] Write protecting the kernel read-only data: 28672k [ 3.963656] debug: unmapping init [mem 0xffffffff9f203000-0xffffffff9f3fffff] [ 3.967083] debug: unmapping init [mem 0xffffffff9fb14000-0xffffffff9fbfffff] [ 4.019145] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.030473] systemd[1]: Detected virtualization kvm. [ 4.032687] systemd[1]: Detected architecture x86-64. [ 4.034718] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.059986] systemd[1]: No hostname configured. [ 4.061591] systemd[1]: Set hostname to . [ 4.063540] random: systemd: uninitialized urandom read (16 bytes read) [ 4.065626] systemd[1]: Initializing machine ID from random generator. [ 4.127796] random: ln: uninitialized urandom read (6 bytes read) [ 4.225574] random: systemd: uninitialized urandom read (16 bytes read) [ 4.227890] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.232254] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.238868] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.935063] device-mapper: uevent: version 1.0.3 [ 4.937782] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.778755] virtio_net virtio0 ens2: renamed from eth0 [ 5.819519] random: fast init done [ 5.903488] scsi host0: ata_piix [ 5.944072] scsi host1: ata_piix [ 5.945824] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.948847] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.487479] random: crng init done [ 10.493947] random: 7 urandom warning(s) missed due to ratelimiting [ 10.785973] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 12.951691] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.203301] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.139894] SELinux: Disabled at runtime. [ 17.325314] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.336949] systemd[1]: Detected virtualization kvm. [ 17.341126] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.799686] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.807263] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.818780] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.830515] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.842847] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.861823] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.871660] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target RPC Port Mapper. [ OK [0[ 20.107448] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS m] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 21.891648] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.850679] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.116421] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 23.782304] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 23.884882] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (10s / no limit)[ 30.627133] Key type dns_resolver registered [ *] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit)[ 31.553339] NFS: Registering the id_resolver key type [ 31.560646] Key type id_resolver registered [ 31.562475] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg423-server login: [ 58.347019] hrtimer: interrupt took 3010867 ns [ 104.919164] libcfs: loading out-of-tree module taints kernel. [ 104.955893] Key type ._llcrypt registered [ 104.965652] Key type .llcrypt registered [ 105.160449] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 126.552737] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 127.756574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 127.780778] alg: No test for adler32 (adler32-zlib) [ 129.341940] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 130.430927] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 132.256303] Key type lgssc registered [ 134.681120] Lustre: Echo OBD driver; http://www.lustre.org/ [ 151.417777] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 205.670663] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 224.304340] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 224.386371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 225.831223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 225.955545] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 226.217692] Lustre: lustre-MDT0000: new disk, initializing [ 226.447719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 226.492939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 232.367625] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.248587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 249.393114] Lustre: 6518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 249.445149] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 249.450132] Lustre: Skipped 1 previous similar message [ 249.595496] Lustre: lustre-MDT0001: new disk, initializing [ 249.707801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 249.790726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 249.809156] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 254.522414] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 260.277520] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 271.303035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 271.568422] Lustre: lustre-OST0000: new disk, initializing [ 271.571745] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 271.577217] Lustre: 8455:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 271.674835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 274.955880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 274.968076] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 275.100777] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 278.982893] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 297.327121] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 297.532211] Lustre: lustre-OST0001: new disk, initializing [ 297.535578] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 297.540748] Lustre: 9530:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 297.608052] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 303.122713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 303.132257] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 303.251171] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 305.709140] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 318.569840] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 325.940923] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 334.120582] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing check_logdir /tmp/testlogs/ [ 340.361940] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing yml_node [ 345.432377] Lustre: DEBUG MARKER: Client: 2.17.57.83 [ 348.767823] Lustre: DEBUG MARKER: MDS: 2.17.57.83 [ 353.227908] Lustre: DEBUG MARKER: OSS: 2.17.57.83 [ 355.377302] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 13:54:31 EDT 2026 [ 379.405523] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 381.245350] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 398.311469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 398.328326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 398.358583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 400.376499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 400.378748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 400.386641] Lustre: Skipped 1 previous similar message [ 400.418950] Lustre: Skipped 1 previous similar message [ 404.693676] Lustre: server umount lustre-MDT0000 complete [ 409.549455] LustreError: 6512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853328 with bad export cookie 4982200356908991943 [ 409.588495] LustreError: 6512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 409.606479] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 410.593832] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 410.594409] LustreError: 10356:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 410.594880] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 410.594884] Lustre: Skipped 2 previous similar messages [ 410.618302] Lustre: Skipped 2 previous similar messages [ 410.630266] LustreError: 10356:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 414.626615] LustreError: 6526:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 414.627412] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 414.650145] LustreError: 6526:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 414.691854] Lustre: Skipped 2 previous similar messages [ 416.035284] Lustre: server umount lustre-MDT0001 complete [ 420.855538] Lustre: server umount lustre-OST0000 complete [ 425.898496] Lustre: server umount lustre-OST0001 complete [ 433.982319] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 447.190952] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 487.866359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 488.203385] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 488.243615] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 488.360368] Lustre: lustre-MDT0000: new disk, initializing [ 488.443024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 488.464897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 494.095368] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 506.271516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 506.400883] Lustre: 15086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 506.462181] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 506.467632] Lustre: Skipped 1 previous similar message [ 506.552745] Lustre: lustre-MDT0001: new disk, initializing [ 506.656807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 506.691262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 506.701983] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 511.186249] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 521.341313] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 528.293596] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 535.844286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 536.142278] Lustre: lustre-OST0000: new disk, initializing [ 536.146258] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 536.152900] Lustre: 17055:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 536.252604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 537.436794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 537.458503] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 537.529763] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 542.431132] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 552.664316] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 558.695924] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 558.958841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 564.054172] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 564.463091] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 568.294396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 568.300982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 568.312682] LustreError: Skipped 1 previous similar message [ 568.317796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 568.340785] Lustre: Skipped 1 previous similar message [ 568.378339] Lustre: Skipped 1 previous similar message [ 578.535038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 578.561273] Lustre: Skipped 3 previous similar messages [ 581.088140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 581.736166] Lustre: server umount lustre-OST0000 complete [ 588.770533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 588.778087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 595.564057] Lustre: server umount lustre-MDT0000 complete [ 599.017500] LustreError: 15099:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 599.034735] LustreError: 15099:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 599.841745] LustreError: 15081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853518 with bad export cookie 4982200356908993476 [ 599.843434] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 599.856229] LustreError: 15081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 600.167654] Lustre: server umount lustre-MDT0001 complete [ 607.271295] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 13:58:43 (1787853523) [ 613.730524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 614.496914] LustreError: 19322:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 614.599065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 618.694141] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 620.001882] LustreError: 19323:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 628.989722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 635.750852] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 647.645333] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 654.525655] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 662.675767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 670.891639] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 678.908136] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 688.603169] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 700.398866] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 700.415976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 700.428456] Lustre: Skipped 3 previous similar messages [ 700.438382] Lustre: Skipped 6 previous similar messages [ 702.270344] Lustre: server umount lustre-MDT0000 complete [ 705.511149] LustreError: 20050:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 705.532272] LustreError: 20050:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 706.480259] LustreError: 19303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853625 with bad export cookie 4982200356908994358 [ 706.495596] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 706.508614] LustreError: 19303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 706.857787] Lustre: server umount lustre-MDT0001 complete [ 722.455446] Lustre: server umount lustre-OST0000 complete [ 750.953633] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 763.836865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 764.498782] LustreError: 23741:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 764.569384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 764.579299] Lustre: Skipped 2 previous similar messages [ 769.679971] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 780.303339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 786.387401] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 790.011201] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 798.375771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 806.913684] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 819.392888] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 819.666571] Lustre: lustre-OST0001: new disk, initializing [ 819.669363] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 819.673505] Lustre: 26057:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 821.049378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 821.059964] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 821.254531] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 828.793242] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 840.885398] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 845.885913] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 862.182166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 862.204212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 862.210389] Lustre: Skipped 2 previous similar messages [ 866.089186] Lustre: server umount lustre-MDT0000 complete [ 867.307596] LustreError: 23736:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 867.330724] LustreError: 23736:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 870.812665] LustreError: 23722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853789 with bad export cookie 4982200356908995205 [ 870.815721] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 870.829310] LustreError: 23722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 871.426954] Lustre: server umount lustre-MDT0001 complete [ 886.981922] Lustre: server umount lustre-OST0000 complete [ 887.776551] Lustre: 3651:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787853790/real 1787853790] req@ffff94b5c3e2f0c0 x1874699797924736/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787853806 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 890.850306] Lustre: 3652:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787853793/real 1787853793] req@ffff94b5c213a580 x1874699797924992/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787853809 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 892.021494] Lustre: server umount lustre-OST0001 complete [ 911.707854] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 924.653507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 925.415693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 925.422069] Lustre: Skipped 3 previous similar messages [ 930.865783] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 934.887661] LustreError: 29078:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 934.913251] LustreError: 29078:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 943.676385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 949.751541] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 953.944769] Lustre: 30227:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 961.775876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 970.643310] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 980.811547] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 987.964623] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 996.301833] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1000.363208] Lustre: 32109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1010.663205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1010.683936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1010.704755] Lustre: Skipped 4 previous similar messages [ 1010.727845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1014.271478] Lustre: server umount lustre-MDT0000 complete [ 1018.505047] LustreError: 29059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853937 with bad export cookie 4982200356908996500 [ 1018.513481] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1018.524963] LustreError: 29059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1019.071846] Lustre: server umount lustre-MDT0001 complete [ 1032.611333] Lustre: server umount lustre-OST0000 complete [ 1047.318211] Lustre: server umount lustre-OST0001 complete [ 1057.602547] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 14:06:13 (1787853973) [ 1087.368473] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1097.177333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1097.533767] LustreError: 34917:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1097.547708] LustreError: 34917:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 12 previous similar messages [ 1101.725944] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1111.035697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1115.986810] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1118.831577] Lustre: 36066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1125.324628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1132.285993] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1140.567482] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1146.948613] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1155.090994] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1158.758075] Lustre: 37943:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1163.748485] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1163.761062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1163.775610] Lustre: Skipped 5 previous similar messages [ 1163.795281] Lustre: Skipped 7 previous similar messages [ 1170.038424] Lustre: server umount lustre-MDT0000 complete [ 1173.825072] LustreError: 34898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854092 with bad export cookie 4982200356908997606 [ 1173.838708] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1173.840615] LustreError: 34898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1174.157819] Lustre: server umount lustre-MDT0001 complete [ 1180.496590] Lustre: server umount lustre-OST0000 complete [ 1184.652964] Lustre: server umount lustre-OST0001 complete [ 1193.577089] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 14:08:30 (1787854110) [ 1209.229632] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1219.406265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1220.012813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1220.019218] Lustre: Skipped 7 previous similar messages [ 1225.364647] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1234.463878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1239.064660] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1242.231665] Lustre: 41232:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1249.630181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1256.867879] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1265.652715] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1273.417662] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1282.096846] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1285.899924] Lustre: 43110:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1286.930712] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1286.939579] Lustre: Skipped 1 previous similar message [ 1291.233215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1291.753291] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1291.781832] Lustre: Skipped 6 previous similar messages [ 1296.426095] Lustre: server umount lustre-MDT0000 complete [ 1300.774635] LustreError: 40065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854219 with bad export cookie 4982200356908998712 [ 1300.788511] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1300.793309] LustreError: 40065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1315.298039] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1315.589165] Lustre: server umount lustre-MDT0001 complete [ 1324.162136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1325.538913] LustreError: 3649:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b6fad47480 x1874699798078080/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1330.783970] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1340.604904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1342.218741] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1346.061713] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1353.590197] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1355.277573] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1361.515531] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1363.043641] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1374.032391] Lustre: server umount lustre-MDT0000 complete [ 1377.656377] LustreError: 41235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854296 with bad export cookie 4982200356908999804 [ 1377.662180] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1377.670753] LustreError: 41235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1377.954849] Lustre: server umount lustre-MDT0001 complete [ 1391.760379] Lustre: server umount lustre-OST0000 complete [ 1393.632491] Lustre: 3652:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787854296/real 1787854296] req@ffff94b6ff8fed00 x1874699798113920/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787854312 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1395.733660] Lustre: server umount lustre-OST0001 complete [ 1404.339263] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 14:12:00 (1787854320) [ 1420.690542] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1431.737199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1432.318536] LustreError: 47721:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1432.346151] LustreError: 47721:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 31 previous similar messages [ 1437.921792] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1446.537078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1451.540616] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1454.838444] Lustre: 48874:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1461.399592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1467.694985] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1476.585502] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1482.279086] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1489.665308] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1494.802876] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1513.443485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1513.457068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1513.462612] Lustre: Skipped 26 previous similar messages [ 1516.061566] Lustre: server umount lustre-MDT0000 complete [ 1520.590431] LustreError: 47702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854439 with bad export cookie 4982200356909000630 [ 1520.598142] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1520.604260] LustreError: 47702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1521.228040] Lustre: server umount lustre-MDT0001 complete [ 1534.700686] Lustre: server umount lustre-OST0000 complete [ 1549.335471] Lustre: server umount lustre-OST0001 complete [ 1559.398435] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 14:14:35 (1787854475) [ 1561.776706] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1563.508203] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 14:14:40 (1787854480) [ 1574.683370] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 1582.873952] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1606.386813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1606.761455] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1606.849954] Lustre: lustre-MDT0000: new disk, initializing [ 1607.000534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1612.095785] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1622.552564] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1630.356367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1630.718798] Lustre: lustre-OST0000: new disk, initializing [ 1630.734318] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1630.747659] Lustre: Skipped 1 previous similar message [ 1630.752525] Lustre: 54952:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1631.942396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1631.957370] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1632.020890] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1637.941645] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1649.261377] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1651.778417] Lustre: DEBUG MARKER: create test files [ 1693.666894] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1693.678446] Lustre: Skipped 11 previous similar messages [ 1697.649876] Lustre: server umount lustre-OST0000 complete [ 1702.071758] Lustre: server umount lustre-MDT0000 complete [ 1714.068575] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 1716.885393] Key type lgssc unregistered [ 1717.179898] LNet: 56586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1717.188090] LNetError: 56586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1717.198804] LNet: Removed LNI 192.168.204.123@tcp [ 1718.174566] Key type .llcrypt unregistered [ 1718.183950] Key type ._llcrypt unregistered [ 1747.910720] Key type ._llcrypt registered [ 1747.913316] Key type .llcrypt registered [ 1748.005748] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1748.931098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1749.015172] alg: No test for adler32 (adler32-zlib) [ 1750.156646] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 1750.557586] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 1752.264319] Key type lgssc registered [ 1753.380947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1761.581783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1761.606298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1763.071403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1767.173613] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1774.501221] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1781.316372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1781.722916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1787.894235] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1787.988150] Lustre: 59321:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1788.002920] Lustre: 59321:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1788.016302] Lustre: 59321:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1788.022965] Lustre: 59321:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1788.034377] Lustre: 59321:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1788.042765] Lustre: 59321:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1788.201598] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1797.589500] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1803.487662] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1810.914255] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1823.724915] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1823.728562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1823.752917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1825.812162] Lustre: server umount lustre-OST0000 complete [ 1830.360764] Lustre: server umount lustre-MDT0000 complete [ 1842.329327] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 1845.117973] Key type lgssc unregistered [ 1845.408716] LNet: 60939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1845.420923] LNetError: 60939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1845.440737] LNet: Removed LNI 192.168.204.123@tcp [ 1846.420108] Key type .llcrypt unregistered [ 1846.427241] Key type ._llcrypt unregistered [ 1875.890908] Key type ._llcrypt registered [ 1875.893334] Key type .llcrypt registered [ 1876.008809] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1877.182872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1877.232103] alg: No test for adler32 (adler32-zlib) [ 1878.305476] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 1878.505817] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 1880.168175] Key type lgssc registered [ 1881.233655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1889.113505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1889.131349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1890.594912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1894.854695] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1902.811763] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1909.915302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1910.197925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1916.348878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1916.459604] Lustre: 63776:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1916.472098] Lustre: 63776:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1916.482842] Lustre: 63776:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1916.496263] Lustre: 63776:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1916.510178] Lustre: 63776:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1916.516311] Lustre: 63776:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1916.788553] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 1924.455723] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1931.410376] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1972.705573] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1972.723366] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1972.751726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1976.337706] Lustre: server umount lustre-OST0000 complete [ 1980.353055] Lustre: server umount lustre-MDT0000 complete [ 1992.036938] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 1994.507295] Key type lgssc unregistered [ 1994.820715] LNet: 65364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1994.832440] LNetError: 65364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1994.863516] LNet: Removed LNI 192.168.204.123@tcp [ 1995.700384] Key type .llcrypt unregistered [ 1995.702791] Key type ._llcrypt unregistered [ 2014.314254] Key type ._llcrypt registered [ 2014.316751] Key type .llcrypt registered [ 2014.429919] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 2030.776728] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2031.794297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2031.817858] alg: No test for adler32 (adler32-zlib) [ 2032.989684] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 2033.273740] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 2034.944194] Key type lgssc registered [ 2036.053498] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2073.127054] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2073.170673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2074.435437] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2074.480978] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2074.590682] Lustre: lustre-MDT0000: new disk, initializing [ 2074.680548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2074.708397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2079.354515] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2091.902522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2092.054638] Lustre: 68520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2092.106525] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2092.115483] Lustre: Skipped 1 previous similar message [ 2092.232539] Lustre: lustre-MDT0001: new disk, initializing [ 2092.394077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2092.472993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2092.499322] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2097.271882] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2107.020253] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2113.407368] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2121.791545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2122.200902] Lustre: lustre-OST0000: new disk, initializing [ 2122.247921] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2122.265456] Lustre: 70487:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2122.385448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2127.936452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2127.948641] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2128.107085] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2129.172301] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2139.609889] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2144.501985] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2144.733451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2149.205148] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2149.400833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2153.955309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2153.955992] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2153.965438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2157.290621] Lustre: server umount lustre-OST0000 complete [ 2164.196951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2164.198618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2164.218932] Lustre: Skipped 1 previous similar message [ 2164.223249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2164.223258] Lustre: Skipped 1 previous similar message [ 2164.244170] LustreError: Skipped 1 previous similar message [ 2168.212515] Lustre: server umount lustre-MDT0000 complete [ 2172.823357] LustreError: 68512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855091 with bad export cookie 14269969110644202632 [ 2172.825471] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2172.840934] LustreError: 68512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2173.174482] Lustre: server umount lustre-MDT0001 complete [ 2182.883583] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 14:24:58 (1787855098) [ 2195.070424] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2233.253851] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 2248.008462] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2287.283497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2287.515655] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2287.565667] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2287.668429] Lustre: lustre-MDT0000: new disk, initializing [ 2287.765817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2287.785693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2292.037075] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2302.118485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2302.244768] Lustre: 76528:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2302.285386] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2302.294409] Lustre: Skipped 1 previous similar message [ 2302.402959] Lustre: lustre-MDT0001: new disk, initializing [ 2302.534666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2302.593864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2302.599329] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2308.517674] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2318.787774] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2324.919439] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2331.482972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2331.710431] Lustre: lustre-OST0000: new disk, initializing [ 2331.713704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2331.719090] Lustre: 78495:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2331.786580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2338.420811] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2340.454386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2340.467578] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2340.581865] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2349.403844] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2354.421204] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2354.645969] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2359.170989] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2359.370245] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2361.320627] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2361.323407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2361.330336] LustreError: Skipped 1 previous similar message [ 2361.337772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2361.352150] Lustre: Skipped 2 previous similar messages [ 2361.356501] Lustre: Skipped 1 previous similar message [ 2366.444234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2366.449219] Lustre: Skipped 2 previous similar messages [ 2367.371729] Lustre: server umount lustre-OST0000 complete [ 2371.562395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2371.564036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2371.582833] Lustre: Skipped 1 previous similar message [ 2371.591117] Lustre: Skipped 1 previous similar message [ 2376.947263] Lustre: server umount lustre-MDT0000 complete [ 2380.407867] LustreError: 76518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855298 with bad export cookie 14269969110644203514 [ 2380.417948] LustreError: 76518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2380.423996] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2380.681974] Lustre: server umount lustre-MDT0001 complete [ 2388.501798] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 14:28:24 (1787855304) [ 2394.745796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2395.143600] LustreError: 80911:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2395.168198] LustreError: 80911:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2395.232324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2399.421371] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2400.229296] LustreError: 80912:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2405.347229] LustreError: 80911:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2407.828333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2412.655577] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2420.331993] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2426.371292] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2432.670580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2432.992583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2432.996953] Lustre: Skipped 1 previous similar message [ 2438.941400] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2446.767650] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2450.249939] Lustre: 83581:0:(genops.c:1774:obd_export_evict_by_uuid()) MGS: evicting 853472e4-8d43-4e31-8b00-854123b9a3b4 at adminstrative request [ 2450.407885] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2450.422689] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2459.619277] Lustre: MGS: Client 853472e4-8d43-4e31-8b00-854123b9a3b4 (at 0@lo) reconnecting [ 2459.634562] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0xc6091d4cf3bea36c to 0xc6091d4cf3bea60c [ 2459.653645] Lustre: MGC192.168.204.123@tcp: Connection restored to 0@lo (at 0@lo) [ 2489.352950] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2489.579806] Lustre: lustre-OST0001: new disk, initializing [ 2489.584238] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2489.590840] Lustre: 83979:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2496.086526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2496.099678] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2496.175384] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2496.750890] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2507.149659] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2511.854501] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2511.855616] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2511.861944] LustreError: Skipped 1 previous similar message [ 2511.868057] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2511.891800] Lustre: Skipped 1 previous similar message [ 2511.906783] Lustre: Skipped 2 previous similar messages [ 2516.166882] Lustre: server umount lustre-OST0001 complete [ 2516.985539] LustreError: 82733:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2517.013384] LustreError: 82733:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2522.103620] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2522.121198] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2522.148316] LustreError: 82734:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.186949] LustreError: 82734:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2532.323884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2532.336019] LustreError: 82732:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2532.346908] Lustre: Skipped 5 previous similar messages [ 2532.377680] LustreError: 82732:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2534.368303] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2534.634497] Lustre: server umount lustre-OST0000 complete [ 2541.539229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2541.548599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2541.567893] Lustre: Skipped 1 previous similar message [ 2546.577910] Lustre: server umount lustre-MDT0000 complete [ 2550.101464] LustreError: 80893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855468 with bad export cookie 14269969110644205068 [ 2550.123429] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2550.625566] Lustre: server umount lustre-MDT0001 complete [ 2559.425079] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 14:31:15 (1787855475) [ 2561.170332] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2563.139218] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 14:31:19 (1787855479) [ 2564.679894] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2566.589703] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 14:31:22 (1787855482) [ 2568.544347] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2570.455318] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 14:31:26 (1787855486) [ 2572.078760] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2574.065814] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 14:31:30 (1787855490) [ 2575.885492] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2583.611961] loop: module loaded [ 2583.635738] loop0: detected capacity change from 0 to 4294967296 [ 2583.736073] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2585.337631] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2589.436805] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2594.901369] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2629.567582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2629.672885] LustreError: 88108:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2629.687503] LustreError: 88108:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2629.702720] LustreError: 88108:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2629.717176] LustreError: 88108:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2655.164911] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 2667.102482] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2713.609565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2713.938616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2713.968856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2714.049974] Lustre: lustre-MDT0000: new disk, initializing [ 2714.165268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2714.173173] Lustre: Skipped 1 previous similar message [ 2714.197562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2720.726907] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2735.616677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2735.798463] Lustre: 90980:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2736.336422] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2742.148803] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2753.572083] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2760.323886] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2768.781550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2769.069155] Lustre: 92952:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2775.836111] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2776.652941] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2776.849334] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2787.273312] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2792.638123] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2792.929724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2798.552516] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2798.902460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2802.661880] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2802.664704] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2802.673869] LustreError: Skipped 1 previous similar message [ 2802.685504] Lustre: Skipped 1 previous similar message [ 2802.690502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2802.700278] Lustre: Skipped 4 previous similar messages [ 2807.317598] Lustre: server umount lustre-OST0000 complete [ 2817.978304] Lustre: server umount lustre-MDT0000 complete [ 2818.032776] LustreError: 90991:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2818.050963] LustreError: 90991:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2822.251843] LustreError: 90970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855740 with bad export cookie 14269969110644205684 [ 2822.258328] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2822.263228] LustreError: 90970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2822.764152] Lustre: server umount lustre-MDT0001 complete [ 2833.143139] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 14:35:49 (1787855749) [ 2841.010067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2847.571687] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2851.825216] LustreError: 95363:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2851.856434] LustreError: 95363:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2858.565887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2859.007560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2859.018826] Lustre: Skipped 3 previous similar messages [ 2864.474415] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2873.982514] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2880.733993] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2889.005124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2896.368443] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2905.193363] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2911.433830] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2912.951793] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2918.264658] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2918.473604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2925.132877] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2925.343507] Lustre: lustre-OST0001: new disk, initializing [ 2925.349185] Lustre: Skipped 2 previous similar messages [ 2925.352741] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2925.357456] Lustre: Skipped 3 previous similar messages [ 2925.361583] Lustre: 98743:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2932.367399] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 2933.402505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2933.422450] Lustre: Skipped 2 previous similar messages [ 2933.433053] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2933.581135] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2943.520449] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2948.863317] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2949.234559] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2955.854472] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2956.301731] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2977.122468] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3008.812462] Lustre: Failing over lustre-MDT0000 [ 3009.074640] Lustre: server umount lustre-MDT0000 complete [ 3010.056445] LustreError: 95360:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3010.085128] LustreError: 95360:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3010.537796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3010.562735] Lustre: Skipped 3 previous similar messages [ 3026.912146] Lustre: 66740:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787855929/real 1787855929] req@000000000940d24b x1874701793383936/t0(0) o400->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787855945 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3026.950471] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3037.743395] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 3037.747836] LDISKFS-fs (dm-0): recovery complete [ 3037.754238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3051.524394] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc6091d4cf3bf7087 [ 3051.537038] Lustre: MGC192.168.204.123@tcp: Connection restored to 0@lo (at 0@lo) [ 3051.927663] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3056.101168] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3056.110700] Lustre: 95358:0:(ldlm_lib.c:2120:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 3056.761854] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3057.137640] Lustre: 96091:0:(ldlm_lib.c:2120:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 3057.154425] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3057.155109] Lustre: 96091:0:(ldlm_lib.c:2120:extend_recovery_timer()) Skipped 1 previous similar message [ 3057.240913] LustreError: 101609:0:(ldlm_lib.c:2733:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3057.570723] LustreError: 101609:0:(ldlm_lib.c:2733:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3057.904135] LustreError: 101609:0:(ldlm_lib.c:2733:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3057.918647] LustreError: 101609:0:(ldlm_lib.c:2733:replay_request_or_update()) Skipped 1 previous similar message [ 3058.264166] LustreError: 101609:0:(ldlm_lib.c:2733:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3058.283245] LustreError: 101609:0:(ldlm_lib.c:2733:replay_request_or_update()) Skipped 1 previous similar message [ 3058.292875] Lustre: 101609:0:(ldlm_lib.c:2120:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3058.310684] Lustre: 101609:0:(ldlm_lib.c:2120:extend_recovery_timer()) Skipped 7 previous similar messages [ 3058.961379] LustreError: 101609:0:(ldlm_lib.c:2733:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3058.968500] LustreError: 101609:0:(ldlm_lib.c:2733:replay_request_or_update()) Skipped 2 previous similar messages [ 3059.288182] LustreError: 101609:0:(ldlm_lib.c:2733:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3059.302715] LustreError: 101609:0:(ldlm_lib.c:2733:replay_request_or_update()) Skipped 2 previous similar messages [ 3060.588886] Lustre: 101609:0:(ldlm_lib.c:2120:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3060.603499] Lustre: 101609:0:(ldlm_lib.c:2120:extend_recovery_timer()) Skipped 6 previous similar messages [ 3061.245920] LustreError: 101609:0:(ldlm_lib.c:2733:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3061.255057] LustreError: 101609:0:(ldlm_lib.c:2733:replay_request_or_update()) Skipped 6 previous similar messages [ 3061.576209] LustreError: 101609:0:(ldlm_lib.c:2733:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3061.589151] LustreError: 101609:0:(ldlm_lib.c:2733:replay_request_or_update()) Skipped 6 previous similar messages [ 3064.786187] Lustre: 101609:0:(ldlm_lib.c:2120:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3064.813089] Lustre: 101609:0:(ldlm_lib.c:2120:extend_recovery_timer()) Skipped 12 previous similar messages [ 3065.458306] LustreError: 101609:0:(ldlm_lib.c:2733:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3065.467309] LustreError: 101609:0:(ldlm_lib.c:2733:replay_request_or_update()) Skipped 12 previous similar messages [ 3065.784336] LustreError: 101609:0:(ldlm_lib.c:2733:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3065.793929] LustreError: 101609:0:(ldlm_lib.c:2733:replay_request_or_update()) Skipped 12 previous similar messages [ 3073.073330] Lustre: 101609:0:(ldlm_lib.c:2120:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3073.096820] Lustre: 101609:0:(ldlm_lib.c:2120:extend_recovery_timer()) Skipped 25 previous similar messages [ 3073.745056] LustreError: 101609:0:(ldlm_lib.c:2733:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3073.755047] LustreError: 101609:0:(ldlm_lib.c:2733:replay_request_or_update()) Skipped 25 previous similar messages [ 3074.080135] LustreError: 101609:0:(ldlm_lib.c:2733:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3074.092400] LustreError: 101609:0:(ldlm_lib.c:2733:replay_request_or_update()) Skipped 25 previous similar messages [ 3089.113431] Lustre: 101609:0:(ldlm_lib.c:2120:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3089.140503] Lustre: 101609:0:(ldlm_lib.c:2120:extend_recovery_timer()) Skipped 49 previous similar messages [ 3089.788389] LustreError: 101609:0:(ldlm_lib.c:2733:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3089.798533] LustreError: 101609:0:(ldlm_lib.c:2733:replay_request_or_update()) Skipped 49 previous similar messages [ 3090.126052] LustreError: 101609:0:(ldlm_lib.c:2733:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3090.136541] LustreError: 101609:0:(ldlm_lib.c:2733:replay_request_or_update()) Skipped 49 previous similar messages [ 3116.304786] Lustre: 101609:0:(ldlm_lib.c:2429:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 3116.324063] Lustre: 101609:0:(ldlm_lib.c:2439:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3116.340969] Lustre: 101609:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client ea205e8f-b152-42ac-870c-1ce8c425f78f@192.168.204.23@tcp [ 3116.356579] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3116.363402] Lustre: 101609:0:(ldlm_lib.c:1949:abort_req_replay_queue()) @@@ aborted: req@00000000eaf3d85f x1874701776395392/t0(8589934781) o101->ea205e8f-b152-42ac-870c-1ce8c425f78f@192.168.204.23@tcp:190/0 lens 592/0 e 0 to 0 dl 1787856045 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 3116.427308] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3116.451070] Lustre: lustre-MDT0000: Denying connection for new client ea205e8f-b152-42ac-870c-1ce8c425f78f (at 192.168.204.23@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 3116.467900] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3116.532722] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3116.546748] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 3116.553472] Lustre: Skipped 2 previous similar messages [ 3116.591548] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:48 to 0x2c0000401:545) [ 3116.601231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:80 to 0x280000401:577) [ 3138.021340] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3138.039322] LustreError: Skipped 1 previous similar message [ 3138.063682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3138.072705] Lustre: Skipped 2 previous similar messages [ 3139.069350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3139.095988] Lustre: Skipped 4 previous similar messages [ 3151.328220] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3151.447661] Lustre: server umount lustre-OST0000 complete [ 3154.407673] LustreError: 98304:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3154.434152] LustreError: 98304:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 57 previous similar messages [ 3161.921686] Lustre: server umount lustre-OST0001 complete [ 3172.406437] Lustre: server umount lustre-MDT0000 complete [ 3176.033610] LustreError: 98742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856094 with bad export cookie 14269969110644256903 [ 3176.048879] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3176.050450] LustreError: 98742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3176.557355] Lustre: server umount lustre-MDT0001 complete [ 3186.502452] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 14:41:42 (1787856102) [ 3194.920755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3194.932973] Lustre: *** cfs_fail_loc=197, val=0*** [ 3210.529540] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3215.651680] LustreError: 103582:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.123@tcp: failed processing log, type 4: rc = -110 [ 3220.800781] Lustre: *** cfs_fail_loc=197, val=0*** [ 3220.806502] Lustre: Skipped 65 previous similar messages [ 3241.397797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3241.416830] Lustre: Skipped 3 previous similar messages [ 3249.291463] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3257.932124] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3260.012828] Lustre: server umount lustre-OST0000 complete [ 3269.238556] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 14:43:05 (1787856185) [ 3291.752399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3297.303993] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3309.295894] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3313.073797] LustreError: 105233:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3313.083989] LustreError: 105233:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3313.093595] LustreError: 105233:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 3313.530114] Lustre: server umount lustre-MDT0000 complete [ 3352.236850] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 3363.969383] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 3407.115051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3407.417842] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3407.484157] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3407.573267] Lustre: lustre-MDT0000: new disk, initializing [ 3407.744341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3412.561939] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3422.578986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3422.710317] Lustre: 109414:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3422.984640] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3427.895443] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3437.809744] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3444.576872] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3452.352268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3452.664781] Lustre: 111382:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3459.791757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3459.996201] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3470.291042] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3475.408621] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3475.639191] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3481.393446] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3481.734402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3485.665993] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3485.666277] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3485.674467] LustreError: Skipped 4 previous similar messages [ 3485.686220] Lustre: Skipped 4 previous similar messages [ 3485.691649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3485.709754] Lustre: Skipped 13 previous similar messages [ 3490.830673] Lustre: server umount lustre-OST0000 complete [ 3510.240358] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3510.564308] Lustre: server umount lustre-MDT0000 complete [ 3511.270901] LustreError: 109426:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3511.304316] LustreError: 109426:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 10 previous similar messages [ 3514.750174] LustreError: 109405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856433 with bad export cookie 14269969110644267592 [ 3514.751441] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3514.761067] LustreError: 109405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3515.222891] Lustre: server umount lustre-MDT0001 complete [ 3525.737658] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 14:47:22 (1787856442) [ 3536.011135] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 3539.058668] Key type lgssc unregistered [ 3539.449867] LNet: 113876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3539.464891] LNetError: 113876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3539.510146] LNet: Removed LNI 192.168.204.123@tcp [ 3540.477323] Key type .llcrypt unregistered [ 3540.479738] Key type ._llcrypt unregistered [ 3559.850410] Key type ._llcrypt registered [ 3559.853215] Key type .llcrypt registered [ 3560.020271] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 3581.099840] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 3581.938964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3582.111951] alg: No test for adler32 (adler32-zlib) [ 3583.115718] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 3583.327944] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 3585.000180] Key type lgssc registered [ 3585.940236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3633.197776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3633.249240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3634.525501] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3634.578337] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3634.692369] Lustre: lustre-MDT0000: new disk, initializing [ 3634.819878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3634.841969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3639.473150] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3649.347606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3649.616494] Lustre: lustre-OST0000: new disk, initializing [ 3649.620806] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3649.624387] Lustre: Skipped 1 previous similar message [ 3649.628955] Lustre: 118052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3649.694708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3655.761074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3655.773489] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3655.814060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3655.915599] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3666.546352] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3666.742856] Lustre: lustre-OST0001: new disk, initializing [ 3666.750119] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3666.760061] Lustre: 118915:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3666.851389] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3673.995448] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3676.865286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3676.877361] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3676.925117] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3687.395320] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3687.405864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3687.411215] Lustre: Skipped 1 previous similar message [ 3689.356898] Lustre: server umount lustre-MDT0000 complete [ 3695.677719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3700.388384] LustreError: 117238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856618 with bad export cookie 11405057186567622881 [ 3700.398366] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3700.402825] LustreError: 117238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3700.495444] Lustre: server umount lustre-OST0000 complete [ 3704.341277] Lustre: server umount lustre-OST0001 complete [ 3713.481407] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 14:50:29 (1787856629) [ 3739.471761] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 14:50:55 (1787856655) [ 3753.907191] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 3777.548428] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 14:51:33 (1787856693) [ 3820.330444] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 3831.076339] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 3869.423782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3869.833645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3869.904253] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3870.025390] Lustre: lustre-MDT0000: new disk, initializing [ 3870.160295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3870.197153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3875.494942] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3887.095221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3887.271948] Lustre: 126690:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3887.354389] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3887.366167] Lustre: Skipped 1 previous similar message [ 3887.474620] Lustre: lustre-MDT0001: new disk, initializing [ 3887.605586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3887.703832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3887.722568] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3893.062922] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3903.152311] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3909.636169] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3917.193170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3917.422900] Lustre: lustre-OST0000: new disk, initializing [ 3917.426158] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3917.431876] Lustre: 128661:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3917.510686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3924.144515] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3925.134501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3925.150766] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3925.282915] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3934.613922] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3941.095289] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3941.408996] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3947.222382] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3947.418406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3951.084196] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3951.087103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3951.088929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3951.119316] Lustre: Skipped 2 previous similar messages [ 3955.914810] Lustre: server umount lustre-OST0000 complete [ 3961.316765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3961.320905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3961.330184] Lustre: Skipped 1 previous similar message [ 3961.333368] Lustre: Skipped 1 previous similar message [ 3968.374108] Lustre: server umount lustre-MDT0000 complete [ 3972.421241] LustreError: 126683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856890 with bad export cookie 11405057186567624106 [ 3972.423504] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3972.429514] LustreError: 126683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3972.778772] Lustre: server umount lustre-MDT0001 complete [ 3982.396548] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 14:54:58 (1787856898) [ 3989.435034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3990.089540] LustreError: 131076:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3990.140194] LustreError: 131076:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [ 3990.202279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3995.621800] LustreError: 131077:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3996.084698] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 3999.722241] LustreError: 131076:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4004.835805] LustreError: 131077:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4006.709970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4012.070836] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4021.021580] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4027.567230] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4034.770043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4034.996223] LustreError: 132897:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4035.128427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4035.132225] Lustre: Skipped 1 previous similar message [ 4041.866193] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4050.975271] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4059.129565] Lustre: *** cfs_fail_loc=159, val=0*** [ 4075.590769] Lustre: lustre-MDT0000: Client 990fc6bd-4667-4163-8363-8f964ad846d7 (at 192.168.204.23@tcp) reconnecting [ 4077.431349] Lustre: *** cfs_fail_loc=159, val=0*** [ 4077.438576] Lustre: Skipped 5 previous similar messages [ 4093.972172] Lustre: lustre-MDT0000: Client 990fc6bd-4667-4163-8363-8f964ad846d7 (at 192.168.204.23@tcp) reconnecting [ 4101.608049] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4101.621154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4101.648149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4101.659420] Lustre: Skipped 1 previous similar message [ 4103.652216] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4106.616354] Lustre: server umount lustre-OST0000 complete [ 4113.889665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4113.912197] Lustre: Skipped 1 previous similar message [ 4113.918083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4113.922747] Lustre: Skipped 3 previous similar messages [ 4117.385962] Lustre: server umount lustre-MDT0000 complete [ 4119.009377] LustreError: 132080:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4119.029161] LustreError: 132080:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 4122.194239] LustreError: 131057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857040 with bad export cookie 11405057186567624981 [ 4122.205623] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4122.213707] LustreError: 131057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4122.642253] Lustre: server umount lustre-MDT0001 complete [ 4135.129430] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 4138.025800] Key type lgssc unregistered [ 4138.378855] LNet: 134853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4138.387900] LNetError: 134853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4138.409481] LNet: Removed LNI 192.168.204.123@tcp [ 4139.588613] Key type .llcrypt unregistered [ 4139.593606] Key type ._llcrypt unregistered [ 4166.802429] Key type ._llcrypt registered [ 4166.803935] Key type .llcrypt registered [ 4166.921525] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4168.291307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4168.343146] alg: No test for adler32 (adler32-zlib) [ 4169.536487] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 4169.784545] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 4171.490945] Key type lgssc registered [ 4172.674762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4180.668831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4180.694991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4182.153157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4187.123800] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4191.717145] LustreError: 136269:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4191.739093] LustreError: 136269:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 4195.482771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4195.903879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4200.354792] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4208.748518] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4216.400057] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4224.369511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4224.749709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4226.818977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4233.273857] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4241.602375] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4250.855259] Lustre: *** cfs_fail_loc=159, val=0*** [ 4267.038449] Lustre: lustre-MDT0000: Client ce451c46-afae-4d2d-aaff-8b3c92ec890f (at 192.168.204.23@tcp) reconnecting [ 4269.423793] Lustre: *** cfs_fail_loc=159, val=0*** [ 4286.016771] Lustre: lustre-MDT0001: Client ce451c46-afae-4d2d-aaff-8b3c92ec890f (at 192.168.204.23@tcp) reconnecting [ 4287.755154] Lustre: *** cfs_fail_loc=159, val=0*** [ 4287.769861] Lustre: Skipped 4 previous similar messages [ 4303.387667] Lustre: lustre-MDT0001: Client ce451c46-afae-4d2d-aaff-8b3c92ec890f (at 192.168.204.23@tcp) reconnecting [ 4308.783991] Lustre: 139306:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4312.233439] Lustre: *** cfs_fail_loc=159, val=0*** [ 4312.237082] Lustre: Skipped 3 previous similar messages [ 4328.496715] Lustre: lustre-MDT0000: Client 3997010d-0abb-4ed1-b7d9-7acf7fd1fedf (at 192.168.204.23@tcp) reconnecting [ 4330.079421] Lustre: *** cfs_fail_loc=159, val=0*** [ 4330.081443] Lustre: Skipped 14 previous similar messages [ 4346.404515] Lustre: lustre-MDT0000: Client 3997010d-0abb-4ed1-b7d9-7acf7fd1fedf (at 192.168.204.23@tcp) reconnecting [ 4347.827394] Lustre: 139608:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4354.017705] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4354.023054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4354.028155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4354.031981] Lustre: Skipped 1 previous similar message [ 4354.051547] Lustre: Skipped 1 previous similar message [ 4359.144208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4359.157260] Lustre: Skipped 1 previous similar message [ 4359.699629] Lustre: server umount lustre-OST0000 complete [ 4364.261337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4364.272400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4364.276877] Lustre: Skipped 1 previous similar message [ 4364.285881] Lustre: Skipped 1 previous similar message [ 4369.378543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4369.393509] Lustre: Skipped 1 previous similar message [ 4369.638028] Lustre: server umount lustre-MDT0000 complete [ 4373.771684] LustreError: 136249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857292 with bad export cookie 4005019294096072279 [ 4373.774183] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4373.798059] LustreError: 136249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4374.269202] Lustre: server umount lustre-MDT0001 complete [ 4386.104153] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 4388.733418] Key type lgssc unregistered [ 4389.076672] LNet: 140666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4389.084703] LNetError: 140666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4389.102349] LNet: Removed LNI 192.168.204.123@tcp [ 4389.908551] Key type .llcrypt unregistered [ 4389.919383] Key type ._llcrypt unregistered [ 4415.223429] Key type ._llcrypt registered [ 4415.228107] Key type .llcrypt registered [ 4415.353641] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4415.952565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4416.043192] alg: No test for adler32 (adler32-zlib) [ 4417.130725] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 4417.550918] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 4419.280560] Key type lgssc registered [ 4420.743358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4430.837877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4430.864360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4432.431664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4437.316690] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4441.577116] LustreError: 142082:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4441.599330] LustreError: 142082:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 4446.694293] LustreError: 142081:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4446.723084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4447.219510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4452.033544] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4460.089728] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4466.406171] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4473.452244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4473.741703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4477.897505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4479.937784] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4487.105737] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4493.231443] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4497.279563] Lustre: 144665:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4500.406375] Lustre: 144714:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4502.327067] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4503.539588] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4507.894608] Lustre: 144861:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4509.235394] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4510.346620] Lustre: 144960:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4518.888734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4518.889853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4518.891864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4518.928024] Lustre: Skipped 1 previous similar message [ 4520.928897] Lustre: server umount lustre-OST0000 complete [ 4529.123310] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4529.124394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4529.137767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4529.148350] Lustre: Skipped 2 previous similar messages [ 4534.241474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4534.252856] Lustre: Skipped 1 previous similar message [ 4539.361131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4539.364581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4539.389073] Lustre: Skipped 1 previous similar message [ 4539.566886] Lustre: server umount lustre-MDT0000 complete [ 4543.337412] LustreError: 142061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857461 with bad export cookie 15305428715577289755 [ 4543.353409] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4543.357219] LustreError: 142061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4543.683627] Lustre: server umount lustre-MDT0001 complete [ 4554.916507] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 4557.461482] Key type lgssc unregistered [ 4557.818620] LNet: 146024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4557.829257] LNetError: 146024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4557.860560] LNet: Removed LNI 192.168.204.123@tcp [ 4558.792872] Key type .llcrypt unregistered [ 4558.795144] Key type ._llcrypt unregistered [ 4584.890196] Key type ._llcrypt registered [ 4584.895764] Key type .llcrypt registered [ 4585.035535] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4586.505731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4586.544074] alg: No test for adler32 (adler32-zlib) [ 4587.722082] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 4587.936941] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 4589.616204] Key type lgssc registered [ 4590.795725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4599.853921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4599.874443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4601.308795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4605.884682] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4610.530513] LustreError: 147441:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4610.539420] LustreError: 147441:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 4614.785107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4615.134608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4620.186054] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4629.717119] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4636.656836] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4644.049037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4644.583145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4645.635960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4650.584379] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4658.196991] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4659.954356] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4664.926085] Lustre: *** cfs_fail_loc=159, val=0*** [ 4680.212845] Lustre: lustre-MDT0000: Client 30b4956d-d68f-4d24-8100-4c40525bab9f (at 192.168.204.23@tcp) reconnecting [ 4701.155406] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4701.157300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4701.184911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4701.202600] Lustre: Skipped 1 previous similar message [ 4704.794537] Lustre: server umount lustre-OST0000 complete [ 4711.402480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4711.419253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4711.421304] Lustre: Skipped 2 previous similar messages [ 4711.449486] Lustre: Skipped 1 previous similar message [ 4715.331029] Lustre: server umount lustre-MDT0000 complete [ 4716.513358] LustreError: 147441:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4716.548986] LustreError: 147441:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 4719.150850] LustreError: 147423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857637 with bad export cookie 13467864556582012761 [ 4719.166982] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4719.172707] LustreError: 147423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4719.522032] Lustre: server umount lustre-MDT0001 complete [ 4730.813310] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 4734.106694] Key type lgssc unregistered [ 4734.411960] LNet: 151114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4734.424769] LNetError: 151114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4734.461755] LNet: Removed LNI 192.168.204.123@tcp [ 4735.411344] Key type .llcrypt unregistered [ 4735.418487] Key type ._llcrypt unregistered [ 4764.362067] Key type ._llcrypt registered [ 4764.363705] Key type .llcrypt registered [ 4764.532476] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4765.595348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4765.672431] alg: No test for adler32 (adler32-zlib) [ 4766.869650] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 4767.227335] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 4769.000759] Key type lgssc registered [ 4770.360749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4779.381725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4779.396456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4780.842559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4786.126870] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4791.268261] LustreError: 152529:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4791.305637] LustreError: 152529:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 4795.747710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4796.193905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4801.254202] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4810.095370] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4817.436971] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4825.878495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4826.085028] LustreError: 154358:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4826.356551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4831.759514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4834.073145] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 4840.962443] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4842.738942] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4863.325788] Lustre: lustre-OST0000: evicting 84623214-e35a-499f-832e-7f6abf75c4fd (at 192.168.204.23@tcp) by administrative request [ 4863.341918] Lustre: Skipped 1 previous similar message [ 4892.131199] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4892.139950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4892.143638] Lustre: Skipped 1 previous similar message [ 4892.161342] Lustre: Skipped 1 previous similar message [ 4897.253544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4897.258844] Lustre: Skipped 1 previous similar message [ 4898.265347] Lustre: server umount lustre-OST0000 complete [ 4902.375876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4902.382333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4902.395165] Lustre: Skipped 1 previous similar message [ 4907.492246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4907.501901] Lustre: Skipped 2 previous similar messages [ 4908.277565] Lustre: server umount lustre-MDT0000 complete [ 4912.370894] LustreError: 153269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857830 with bad export cookie 13774746728274073709 [ 4912.373341] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4912.388436] LustreError: 153269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4912.804526] Lustre: server umount lustre-MDT0001 complete [ 4926.928995] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 4930.270118] Key type lgssc unregistered [ 4930.741835] LNet: 156600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4930.755262] LNetError: 156600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4930.780549] LNet: Removed LNI 192.168.204.123@tcp [ 4931.714232] Key type .llcrypt unregistered [ 4931.716480] Key type ._llcrypt unregistered [ 5025.403488] Key type ._llcrypt registered [ 5025.409739] Key type .llcrypt registered [ 5025.506393] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 5026.547831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5026.697068] alg: No test for adler32 (adler32-zlib) [ 5027.886061] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 5028.189540] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 5029.864192] Key type lgssc registered [ 5031.077459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5040.128889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5040.164977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5041.782348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5046.838856] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5052.388322] LustreError: 159093:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5052.437483] LustreError: 159093:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5056.639256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5057.072494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5062.006673] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5070.759904] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5077.055147] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5083.712952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5084.085279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5087.162477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 5090.257725] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5099.654845] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5101.493794] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 5111.780536] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5111.786495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5111.802471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5116.909968] LustreError: 160923:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5117.022578] Lustre: server umount lustre-OST0000 complete [ 5122.022140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5122.023048] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5122.035868] Lustre: Skipped 1 previous similar message [ 5122.058468] Lustre: Skipped 2 previous similar messages [ 5127.139217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5127.148188] Lustre: Skipped 2 previous similar messages [ 5127.855855] Lustre: server umount lustre-MDT0000 complete [ 5132.258934] LustreError: 160304:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5132.289070] LustreError: 160304:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5132.407796] LustreError: 159072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858050 with bad export cookie 1451369440298973320 [ 5132.441052] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5132.810544] Lustre: server umount lustre-MDT0001 complete [ 5146.932756] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 5149.552201] Key type lgssc unregistered [ 5149.821358] LNet: 162638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5149.837271] LNetError: 162638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5149.862224] LNet: Removed LNI 192.168.204.123@tcp [ 5150.661074] Key type .llcrypt unregistered [ 5150.664077] Key type ._llcrypt unregistered [ 5181.303555] Key type ._llcrypt registered [ 5181.309787] Key type .llcrypt registered [ 5181.444438] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 5182.523052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5182.584063] alg: No test for adler32 (adler32-zlib) [ 5184.095780] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 5184.455095] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 5186.184211] Key type lgssc registered [ 5187.330733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5205.828769] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 15:15:21 (1787858121) [ 5222.684626] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 5235.159444] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 5279.184635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5279.220608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5280.499713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5280.530245] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5280.610632] Lustre: lustre-MDT0000: new disk, initializing [ 5280.681608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5280.713980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5286.317535] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5300.798162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5300.957499] Lustre: 166907:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5301.026773] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5301.035392] Lustre: Skipped 1 previous similar message [ 5301.190907] Lustre: lustre-MDT0001: new disk, initializing [ 5301.443932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5301.554949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5301.577224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5307.532907] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5320.305217] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5329.029104] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5338.356043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5338.792775] Lustre: lustre-OST0000: new disk, initializing [ 5338.797031] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5338.804711] Lustre: 168874:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5338.934830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5346.315727] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5347.909114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5347.920556] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5348.055199] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5356.936764] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5368.802594] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5368.805525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5368.814530] LustreError: Skipped 1 previous similar message [ 5368.817995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5368.836502] Lustre: Skipped 1 previous similar message [ 5368.857792] Lustre: Skipped 1 previous similar message [ 5373.929528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5373.939455] Lustre: Skipped 1 previous similar message [ 5374.977755] Lustre: server umount lustre-OST0000 complete [ 5379.063954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5379.088993] Lustre: Skipped 1 previous similar message [ 5379.093186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5379.105314] Lustre: Skipped 1 previous similar message [ 5384.163358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5384.168799] Lustre: Skipped 1 previous similar message [ 5389.291827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5389.303751] Lustre: Skipped 1 previous similar message [ 5393.376257] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5393.529947] Lustre: server umount lustre-MDT0000 complete [ 5397.911150] LustreError: 166897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858316 with bad export cookie 9743804739658789247 [ 5397.921396] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5397.930532] LustreError: 166897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5398.386792] Lustre: server umount lustre-MDT0001 complete [ 5413.225149] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 5416.709670] Key type lgssc unregistered [ 5417.069301] LNet: 170786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5417.079174] LNetError: 170786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5417.092032] LNet: Removed LNI 192.168.204.123@tcp [ 5418.025059] Key type .llcrypt unregistered [ 5418.029739] Key type ._llcrypt unregistered [ 5446.067604] Key type ._llcrypt registered [ 5446.069462] Key type .llcrypt registered [ 5446.209978] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 5447.831539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5447.854945] alg: No test for adler32 (adler32-zlib) [ 5448.879816] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 5449.252337] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 5450.976876] Key type lgssc registered [ 5452.429494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5460.768063] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5460.800535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5462.250109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5467.640788] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5471.718587] LustreError: 172205:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5471.735410] LustreError: 172205:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5476.836063] LustreError: 172204:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5481.288656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5481.795737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5488.710130] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5498.838976] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5506.928933] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5515.254793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5515.610454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5522.891837] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 5532.034632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5532.043643] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5540.709734] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5540.734795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5540.754848] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5540.783859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.813817] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5541.942979] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5541.966335] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5541.985565] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5541.994660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.045623] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5543.178625] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5543.215351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5543.237073] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5543.255032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.267660] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5544.556766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.643699] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5545.651858] Lustre: Skipped 1 previous similar message [ 5545.675604] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5545.684680] Lustre: Skipped 1 previous similar message [ 5545.687659] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5545.691288] Lustre: Skipped 1 previous similar message [ 5545.694932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.707409] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5545.712810] Lustre: Skipped 1 previous similar message [ 5546.743534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.993037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.205103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.307936] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5550.318403] Lustre: Skipped 3 previous similar messages [ 5550.343644] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5550.353143] Lustre: Skipped 3 previous similar messages [ 5550.358578] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5550.366971] Lustre: Skipped 3 previous similar messages [ 5550.374682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.389071] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5550.399040] Lustre: Skipped 3 previous similar messages [ 5551.566757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.795184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.663549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.186335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.223548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.543589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.763875] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5558.778525] Lustre: Skipped 6 previous similar messages [ 5558.796647] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5558.807996] Lustre: Skipped 6 previous similar messages [ 5558.812115] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5558.823169] Lustre: Skipped 6 previous similar messages [ 5558.835388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.850348] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5558.855796] Lustre: Skipped 6 previous similar messages [ 5559.988904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.198456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.493849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.659634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.617387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.751658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.785064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.811889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.848742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.100634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.332176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.614368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.748855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.817542] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5574.823862] Lustre: Skipped 13 previous similar messages [ 5574.839600] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5574.846573] Lustre: Skipped 13 previous similar messages [ 5574.849965] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5574.853867] Lustre: Skipped 13 previous similar messages [ 5574.857425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.867392] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5574.879787] Lustre: Skipped 13 previous similar messages [ 5575.841254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.726755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.669052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.701468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.502341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.353449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.464704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.208582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.039668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.846594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.092776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.088065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.890664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.806270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.734346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.500563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.485678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.476844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.946178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.946259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.990796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.200200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.203298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.297485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.343820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.331701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.438771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.233631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.288796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.149553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.321375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.486256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.415676] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5607.424823] Lustre: Skipped 32 previous similar messages [ 5607.439954] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5607.451619] Lustre: Skipped 32 previous similar messages [ 5607.460889] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5607.470113] Lustre: Skipped 32 previous similar messages [ 5607.474627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.486812] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5607.498734] Lustre: Skipped 32 previous similar messages [ 5608.557653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.410527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.571874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.579306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.752592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.697883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.910504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.948914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.990902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.009070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.228699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.187261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.552853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.737512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.936781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.123037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.209835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.249142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.261978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.368750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.347727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.541208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.594254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.679828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.109163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.043051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.116276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.136335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.099298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.981176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.910856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.940124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.883954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.714364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.629706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.505696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.398903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.332512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.238750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.208065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.105474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.017950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.955993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.903866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.808638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.709786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.681877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.640663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.567401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.400820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.375834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.183991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.165935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.212150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.067609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.069566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.948242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.910365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.835183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.822914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.715655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.762062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.851612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.725557] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5671.730160] Lustre: Skipped 63 previous similar messages [ 5671.743895] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5671.751891] Lustre: Skipped 63 previous similar messages [ 5671.755932] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5671.760156] Lustre: Skipped 63 previous similar messages [ 5671.762980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.776131] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5671.782376] Lustre: Skipped 63 previous similar messages [ 5672.637076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.620823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.616599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.398541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.292742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.138950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.037128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.019690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.900218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.720857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.703745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.630505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.675686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.807760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.687037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.805502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.673454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.760290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.621347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.509660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.495084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.632244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.494315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.450669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.402972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.390291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.334822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.341353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.220408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.387855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.287392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.344290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.225243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.156870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.980981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.833323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.637560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.546705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.475888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.333748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.144712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.969837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.711619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.658710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.408971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.367860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.233449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.349760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.311340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.607203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.058535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.095373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.086760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.998478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.108862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.180604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.093119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.214355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.076573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.003582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.922469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.746429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.480767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.422283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.368283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.235771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.101472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.016783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.030586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.037186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.975893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.843481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.779547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.954315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.183892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.136889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.143495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.067911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.046528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.872213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.852471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.855504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.042762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.226720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.138792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.150542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.069536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.005744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.858978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.607578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.358602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.275990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.154511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.067187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.914642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.840566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.803912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.711594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.475745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.333286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.165433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.007764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.837284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.851806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.871119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.737319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.618924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.509920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.388113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.280314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.183426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.201824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.315795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.487496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.390977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.529481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.635133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.680861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.499049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.736757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.962199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.322532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.369924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.587517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.763178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.780265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.835769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.159453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.600979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.872031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.827844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.229829] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5800.236185] Lustre: Skipped 131 previous similar messages [ 5800.260669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5800.277110] Lustre: Skipped 131 previous similar messages [ 5800.281163] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5800.283572] Lustre: Skipped 131 previous similar messages [ 5800.289588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.298768] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5800.301304] Lustre: Skipped 131 previous similar messages [ 5801.643583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.091755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.275688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.172716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.418850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.672520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.996400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.330020] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.650620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.549160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.343510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.450221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.253143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.227433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.322784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.460391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.414049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.469277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.768432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.823148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.842494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.998188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.179795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.177607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.448288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.672614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.830805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.908966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.022941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.069043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.127683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.455338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.398175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.367256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.605373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.895306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.914893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.881920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.804036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.854875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.763134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.727522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.715905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.828389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.087840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.206155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.435478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.606818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.782068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.859825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.913840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.033899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.013769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.060146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.998168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.838613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.779590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.995405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.949827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.935994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.900758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.011635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.822537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.746584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.640743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.749845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.680783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.557119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.544092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.545666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.451758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.415878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.329360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.477624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.427525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.420707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.350440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.316868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.281198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.350517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.207548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.182924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.069716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.069551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.859056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.907416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.887507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.944686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.947583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.025861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.975375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.956727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.863072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.840629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.749254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.660766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.632411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.492910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.455693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.491665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.476612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.675696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.516510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.441259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.573083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.531712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.446265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.354514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.388427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.406715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.323362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.427457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.320302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.211157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.147554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.134094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.175310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.221693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.012849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.824800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.761789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.650640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.510321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.447813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.518615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.397636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.424525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.357302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.380784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.302894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.213743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.153756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.055344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.225163] LustreError: 172224:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5937.243326] LustreError: 172224:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5937.257877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.172039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.132688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.145584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.061192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.872697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.794667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.777645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.758995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.671729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.543767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.583970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.631270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.623905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.497379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.371423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.233434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.119356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.884254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.843848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.766897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.956951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.874335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.832072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.644591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.440991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.291821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.409860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.589177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.393647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.229304] LustreError: 172224:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5965.243369] LustreError: 172224:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5965.252285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.144691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.040688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.898795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.857374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.660626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.565338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.427249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.429431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.498818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.505405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.603835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.532473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.419205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.308113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.162455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.181356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.403779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.639968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.795809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.055188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.919392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.906286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.767822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.684580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.476215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.485590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.335194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.132503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.898399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.646893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.491893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.368126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.168564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.928196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.667783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.443882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.183937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.848215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.504395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.213953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.029079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.884315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.695493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.597243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.257024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.126831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.927667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.752335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.671617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.617716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.546038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.619300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.577413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.566724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.451187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.482563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.448193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.318634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.305550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.206714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.012791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.923669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.882236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.768386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.753735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.917612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.633316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.589185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.567404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.530294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.526727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.373490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.406941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.435543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.196859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.163499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.967105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.154777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.059260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.016396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.885081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.732551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.572337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.548250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.639814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.288827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.649962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.361536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.045118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.914944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.253157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.489294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.707148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.725896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.832385] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6056.846450] Lustre: Skipped 258 previous similar messages [ 6056.881532] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6056.897098] Lustre: Skipped 258 previous similar messages [ 6056.907646] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6056.914953] Lustre: Skipped 258 previous similar messages [ 6056.928611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.960839] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6056.977497] Lustre: Skipped 258 previous similar messages [ 6058.194767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.557191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.490177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.813742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.919986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.150837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.242076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.369325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.393435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.339798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.578672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.751595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.772455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.915376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.979660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.898638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.970751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.915566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.926524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.004460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.946246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.960438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.975481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.109161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.075554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.115579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.079336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.035914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.294480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.722927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.746749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.918429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.969093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.302766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.554720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.848586] LustreError: 172224:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6096.860563] LustreError: 172224:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6096.877455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.747308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.681224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.738738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.760297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.879708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.925980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.246438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.346601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.339533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.522286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.763954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.200109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.421507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.568322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.034211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.309626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.570392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.995864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.271545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.653984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.915929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.871184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.103316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.161700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.341922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.263736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.156484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.182227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.243645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.188886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.121177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.905548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.984310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.159848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.070122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.045210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.929901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.877961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.738915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.660277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.500896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.592506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.583333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.519693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.357251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.415568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.490557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.524647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.485565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.494733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.449667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.349774] LustreError: 172224:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6152.359640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.366457] LustreError: 172224:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6153.420709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6154.292208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.144950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.030404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6263.790603] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6263.811557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6264.290648] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6267.931100] Lustre: server umount lustre-OST0000 complete [ 6274.535043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6274.554870] Lustre: Skipped 3 previous similar messages [ 6278.582825] Lustre: server umount lustre-MDT0000 complete [ 6279.652664] LustreError: 172205:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6279.672784] LustreError: 172205:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6282.830504] LustreError: 172185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859201 with bad export cookie 10375624357354145603 [ 6282.836185] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6282.842453] LustreError: 172185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6283.279479] Lustre: server umount lustre-MDT0001 complete [ 6294.050718] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 6296.660615] Key type lgssc unregistered [ 6296.964653] LNet: 207343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6296.970151] LNetError: 207343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6296.986957] LNet: Removed LNI 192.168.204.123@tcp [ 6297.824426] Key type .llcrypt unregistered [ 6297.826516] Key type ._llcrypt unregistered [ 6322.786871] Key type ._llcrypt registered [ 6322.793271] Key type .llcrypt registered [ 6322.946670] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 6324.093914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6324.172077] alg: No test for adler32 (adler32-zlib) [ 6325.210656] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 6325.387108] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 6327.032329] Key type lgssc registered [ 6328.350514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6336.443196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6336.462174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6337.982296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6342.773399] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6348.264473] LustreError: 208757:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6348.291964] LustreError: 208757:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6351.378775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6351.743831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6356.558521] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6365.282615] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6371.999960] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6381.167895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6381.645377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6382.719078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11443 to 0x280000401:11489) [ 6388.452899] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6396.871489] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6403.190871] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6405.292864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6405.314297] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6405.325112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6405.356298] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6405.359150] LustreError: 208777:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 6410.246457] Lustre: lustre-OST0000: Not available for connect from 192.168.204.23@tcp (stopping) [ 6412.782269] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6412.785474] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6412.803380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6412.825688] Lustre: Skipped 1 previous similar message [ 6414.138786] Lustre: server umount lustre-OST0000 complete [ 6425.442398] LustreError: 208777:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 6448.759849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6449.189838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6453.294930] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6453.318653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6453.325789] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6453.345271] Lustre: Skipped 1 previous similar message [ 6455.784482] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6463.606704] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6465.208636] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6470.302891] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8dc951fd4000.ost_server_uuid 50 [ 6473.172668] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8dc951fd4000.ost_server_uuid in IDLE state after 1 sec [ 6478.820413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6478.823586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6478.825939] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6478.836891] Lustre: Skipped 1 previous similar message [ 6478.866643] Lustre: Skipped 2 previous similar messages [ 6482.983762] Lustre: server umount lustre-OST0000 complete [ 6489.057990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6489.072206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6489.080549] Lustre: Skipped 1 previous similar message [ 6492.946701] Lustre: server umount lustre-MDT0000 complete [ 6494.177144] LustreError: 211163:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6494.190692] LustreError: 211163:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6496.603961] LustreError: 208737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859415 with bad export cookie 8253441615680080465 [ 6496.620930] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6496.623861] LustreError: 208737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6497.029056] Lustre: server umount lustre-MDT0001 complete [ 6509.768461] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 6512.891877] Key type lgssc unregistered [ 6513.217275] LNet: 213971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6513.237863] LNetError: 213971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6513.267190] LNet: Removed LNI 192.168.204.123@tcp [ 6514.389616] Key type .llcrypt unregistered [ 6514.391415] Key type ._llcrypt unregistered [ 6550.070502] Key type ._llcrypt registered [ 6550.073764] Key type .llcrypt registered [ 6550.196490] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 6551.447918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6551.489469] alg: No test for adler32 (adler32-zlib) [ 6552.501783] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 6552.718897] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 6554.386952] Key type lgssc registered [ 6555.610936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6565.614778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6565.656888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6567.374707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6573.423867] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6576.611395] LustreError: 215701:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6576.632869] LustreError: 215701:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6581.736666] LustreError: 215702:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6584.469923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6585.158271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6590.884715] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6593.760452] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6601.907706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6602.308847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6608.787131] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6611.510171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11491 to 0x280000401:11521) [ 6617.060088] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6617.213698] Lustre: lustre-OST0001: new disk, initializing [ 6617.217644] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6617.224462] Lustre: 218014:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6617.328673] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6622.129378] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6626.432211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6626.456582] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6626.554581] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6634.022661] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6639.459337] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6656.997703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6657.011619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6657.041974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6657.511354] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6657.538570] Lustre: Skipped 1 previous similar message [ 6657.548810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6657.555590] Lustre: Skipped 3 previous similar messages [ 6660.405469] Lustre: server umount lustre-MDT0000 complete [ 6662.629249] LustreError: 216429:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6662.643532] LustreError: 216429:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 6664.535144] LustreError: 218896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859583 with bad export cookie 11426411314159408014 [ 6664.548335] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6664.551491] LustreError: 218896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6664.911818] Lustre: server umount lustre-MDT0001 complete [ 6679.807854] Lustre: server umount lustre-OST0000 complete [ 6694.095992] Lustre: server umount lustre-OST0001 complete [ 6718.813794] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 6727.843260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6728.265055] LustreError: 221119:0:(ldlm_lib.c:1200:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6728.342126] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6732.956191] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6733.793584] LustreError: 221120:0:(ldlm_lib.c:1200:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6742.704678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6743.167964] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6747.719650] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6751.473978] Lustre: 222261:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6760.451442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6760.913562] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6768.102172] LustreError: 222941:0:(ldlm_lib.c:1200:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6768.135217] LustreError: 222941:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6768.466195] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6773.245282] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11524 to 0x280000401:11553) [ 6779.706524] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6786.977921] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6795.836983] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6799.871243] Lustre: 224135:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6814.689445] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6814.700382] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6814.718590] Lustre: Skipped 1 previous similar message [ 6814.729624] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6819.377139] Lustre: server umount mylustre-MDT0000 complete [ 6821.348337] LustreError: 222175:0:(ldlm_lib.c:1200:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6821.380531] LustreError: 222175:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 7 previous similar messages [ 6823.464292] LustreError: 224335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859742 with bad export cookie 11426411314159410814 [ 6823.476683] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6823.481773] LustreError: 224335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6823.854282] Lustre: server umount mylustre-MDT0001 complete [ 6838.257985] Lustre: server umount mylustre-OST0000 complete [ 6852.621113] Lustre: server umount mylustre-OST0001 complete [ 6880.610721] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 6894.802211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6895.284817] LustreError: 226447:0:(ldlm_lib.c:1200:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6895.366825] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6895.374269] Lustre: Skipped 1 previous similar message [ 6900.874196] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6909.166490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6913.786446] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6916.561539] Lustre: 227589:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6923.587349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6930.600770] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6940.685780] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11524 to 0x280000401:11585) [ 6941.328533] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6946.845782] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6950.688970] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 6962.005638] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6966.587280] Lustre: 229462:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6975.970279] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6975.980960] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6975.996246] Lustre: Skipped 3 previous similar messages [ 6975.999545] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 6976.002943] Lustre: Skipped 4 previous similar messages [ 6977.617919] Lustre: server umount tfs-MDT0000 complete [ 6981.934834] LustreError: 227591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859900 with bad export cookie 11426411314159412487 [ 6981.946333] LustreError: 227591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6981.954194] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6982.626342] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6982.628089] LustreError: 227963:0:(ldlm_lib.c:1200:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6982.636051] Lustre: Skipped 4 previous similar messages [ 6982.689698] LustreError: 227963:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 9 previous similar messages [ 6982.909691] Lustre: server umount tfs-MDT0001 complete [ 6988.001743] Lustre: server umount tfs-OST0000 complete [ 6993.338720] Lustre: server umount tfs-OST0001 complete [ 7018.790066] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7030.824282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7031.228982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7031.237726] Lustre: Skipped 3 previous similar messages [ 7036.388244] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7044.908036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7049.179056] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7051.890460] Lustre: 232849:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7059.588765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7069.001520] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7075.535675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11524 to 0x280000401:11617) [ 7081.241795] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7086.611947] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7089.231336] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7097.871444] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7102.186263] Lustre: 234725:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7110.278666] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 15:47:06 (1787860026) [ 7127.011263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7127.020578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7127.045689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7127.055322] Lustre: Skipped 4 previous similar messages [ 7137.768672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7137.788781] Lustre: Skipped 10 previous similar messages [ 7141.344858] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7141.490830] Lustre: server umount lustre-MDT0000 complete [ 7142.892951] LustreError: 231709:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7142.911702] LustreError: 231709:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 11 previous similar messages [ 7145.269566] LustreError: 231684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860063 with bad export cookie 11426411314159414153 [ 7145.275885] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7145.279625] LustreError: 231684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7148.003866] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7148.034319] Lustre: Skipped 4 previous similar messages [ 7151.752366] Lustre: server umount lustre-MDT0001 complete [ 7162.502716] Lustre: server umount lustre-OST0000 complete [ 7172.648825] Lustre: server umount lustre-OST0001 complete [ 7180.987874] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 7195.722789] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7237.518506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7237.865498] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7237.911827] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7238.030575] Lustre: lustre-MDT0000: new disk, initializing [ 7238.247279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 7238.264846] Lustre: Skipped 3 previous similar messages [ 7238.300221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7243.443433] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7256.694037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7256.833211] Lustre: 237930:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7256.897615] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7256.903160] Lustre: Skipped 1 previous similar message [ 7256.965409] Lustre: lustre-MDT0001: new disk, initializing [ 7257.094863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7257.122282] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7262.358322] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7273.975860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7274.301898] Lustre: lustre-OST0000: new disk, initializing [ 7274.320641] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7274.330973] Lustre: 239515:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7275.571540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7275.581353] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7275.686939] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7282.228937] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7294.162826] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7294.453912] Lustre: lustre-OST0001: new disk, initializing [ 7294.463964] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7294.483725] Lustre: 240385:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7295.607443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7295.627399] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7295.793280] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7302.181206] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7316.151823] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 15:50:32 (1787860232) [ 7333.902638] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 15:50:50 (1787860250) [ 7352.293915] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7352.314834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7352.322088] Lustre: Skipped 7 previous similar messages [ 7363.040682] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7363.166387] Lustre: server umount lustre-OST0000 complete [ 7373.539246] Lustre: server umount lustre-MDT0000 complete [ 7377.349884] LustreError: 237925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860295 with bad export cookie 11426411314159415707 [ 7377.371667] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7377.375168] LustreError: 237925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7383.865058] Lustre: server umount lustre-MDT0001 complete [ 7395.859771] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 7469.864284] Lustre: server umount lustre-OST0001 complete [ 7477.534467] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 7495.218755] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7539.409109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7539.727373] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7539.771935] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7539.909567] Lustre: lustre-MDT0000: new disk, initializing [ 7540.041651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7540.052884] Lustre: Skipped 3 previous similar messages [ 7540.095506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7545.770076] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7558.735439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7558.871508] Lustre: 245236:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7558.959683] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7558.964053] Lustre: Skipped 1 previous similar message [ 7559.100197] Lustre: lustre-MDT0001: new disk, initializing [ 7559.309580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7559.332475] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7564.545243] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7576.734341] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7583.155642] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7591.487617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7591.789963] Lustre: lustre-OST0000: new disk, initializing [ 7591.794713] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7591.811169] Lustre: 247206:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7593.390407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7593.400225] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7593.543488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7598.834871] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7608.879185] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7619.054799] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7619.063977] Lustre: Skipped 5 previous similar messages [ 7619.067034] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7619.077511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7619.077524] Lustre: Skipped 12 previous similar messages [ 7619.101556] LustreError: Skipped 1 previous similar message [ 7621.480727] Lustre: server umount lustre-OST0000 complete [ 7632.445210] Lustre: server umount lustre-MDT0000 complete [ 7634.402916] LustreError: 246077:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7634.415024] LustreError: 246077:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 11 previous similar messages [ 7636.327486] LustreError: 245227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860554 with bad export cookie 11426411314159417177 [ 7636.330498] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7636.337683] LustreError: 245227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7636.700926] Lustre: server umount lustre-MDT0001 complete [ 7651.141551] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 7655.189108] Key type lgssc unregistered [ 7655.832586] LNet: 248969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7655.846857] LNetError: 248969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7655.881581] LNet: Removed LNI 192.168.204.123@tcp [ 7657.615640] Key type .llcrypt unregistered [ 7657.620319] Key type ._llcrypt unregistered [ 7688.952924] Key type ._llcrypt registered [ 7688.956137] Key type .llcrypt registered [ 7689.070524] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7690.649878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7690.741106] alg: No test for adler32 (adler32-zlib) [ 7691.904806] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 7692.248867] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 7694.032719] Key type lgssc registered [ 7695.402272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7704.658929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7704.687728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7706.306723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7711.353422] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7715.811357] LustreError: 250433:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7715.837310] LustreError: 250433:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7720.751547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7720.938521] LustreError: 250434:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7721.374907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7726.887763] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7735.621809] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7742.810590] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7751.726315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7752.194800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7758.800802] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7767.515567] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7768.653453] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 7770.602530] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7770.603404] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7770.630278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7775.716730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7776.751258] LustreError: 252256:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7776.964262] Lustre: server umount lustre-OST0000 complete [ 7781.857279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7781.892151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7781.893477] Lustre: Skipped 1 previous similar message [ 7781.915146] Lustre: Skipped 1 previous similar message [ 7786.980623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7786.990056] Lustre: Skipped 1 previous similar message [ 7788.040840] Lustre: server umount lustre-MDT0000 complete [ 7792.107994] LustreError: 250433:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7792.143272] LustreError: 250433:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7792.299314] LustreError: 252965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860710 with bad export cookie 18080156511282036228 [ 7792.314405] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7792.317127] LustreError: 252965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7792.725061] Lustre: server umount lustre-MDT0001 complete [ 7805.404074] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 7808.671263] Key type lgssc unregistered [ 7809.022083] LNet: 254022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7809.035271] LNetError: 254022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7809.059825] LNet: Removed LNI 192.168.204.123@tcp [ 7810.342412] Key type .llcrypt unregistered [ 7810.345649] Key type ._llcrypt unregistered [ 7828.810337] Key type ._llcrypt registered [ 7828.817105] Key type .llcrypt registered [ 7828.929642] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7830.258624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7830.405630] alg: No test for adler32 (adler32-zlib) [ 7831.571253] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 7831.944954] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 7833.680213] Key type lgssc registered [ 7834.877888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7845.796592] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7845.826929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7861.600541] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7866.720421] LustreError: 255217:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.123@tcp: failed processing log, type 4: rc = -110 [ 7893.409320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7899.739291] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 7908.061428] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7910.565735] Lustre: server umount lustre-OST0000 complete [ 7930.126938] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 7933.011652] Key type lgssc unregistered [ 7933.286833] LNet: 256853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7933.293201] LNetError: 256853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7933.310811] LNet: Removed LNI 192.168.204.123@tcp [ 7934.261702] Key type .llcrypt unregistered [ 7934.264451] Key type ._llcrypt unregistered [ 7981.587344] Key type ._llcrypt registered [ 7981.590943] Key type .llcrypt registered [ 7981.728466] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 7983.157431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7983.222186] alg: No test for adler32 (adler32-zlib) [ 7984.359225] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 7984.555908] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 7986.250121] Key type lgssc registered [ 7987.737866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8002.714448] loop0: detected capacity change from 0 to 204800000 [ 8003.078757] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8003.101534] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8003.564067] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8004.681843] loop0: detected capacity change from 0 to 204800000 [ 8004.903372] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8004.925622] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8005.507110] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8006.637340] loop0: detected capacity change from 0 to 204800000 [ 8006.846303] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8006.865325] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8006.984725] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8008.267323] loop0: detected capacity change from 0 to 204800000 [ 8008.513936] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8008.530639] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8008.692082] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8010.056473] loop0: detected capacity change from 0 to 204800000 [ 8010.091100] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8015.455177] loop0: detected capacity change from 0 to 204800000 [ 8015.480342] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8020.100636] loop0: detected capacity change from 0 to 204800000 [ 8020.133475] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8025.003496] loop0: detected capacity change from 0 to 204800000 [ 8025.041345] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8030.018516] loop0: detected capacity change from 0 to 204800000 [ 8030.110982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8030.221400] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8030.423368] Lustre: 260283:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8030.449992] Lustre: 260283:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 8030.462055] Lustre: 260283:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8030.480369] Lustre: 260283:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 8030.509257] Lustre: 260283:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8030.536156] Lustre: 260283:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 8030.560780] Lustre: 260283:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8030.582969] Lustre: 260283:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 8030.611762] Lustre: 260283:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 8030.645626] Lustre: 260283:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 8030.665495] Lustre: 260283:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 8030.684564] Lustre: 260283:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 8030.700585] Lustre: 260283:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 8030.722469] Lustre: 260283:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 8036.934907] Lustre: server umount MGS complete [ 8037.842285] loop0: detected capacity change from 0 to 204800000 [ 8037.946874] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8038.370942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8038.408624] Lustre: lustre-MDT0000: Aborting client recovery [ 8039.256492] loop1: detected capacity change from 0 to 204800000 [ 8039.379429] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8039.459868] Lustre: 260715:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8039.470185] Lustre: 260715:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 8039.477502] Lustre: 260715:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 8039.487131] Lustre: 260715:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8039.498777] Lustre: 260715:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8039.507512] Lustre: 260715:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 8039.515210] Lustre: 260715:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8039.539349] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8039.721936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8039.772491] Lustre: lustre-MDT0001: Aborting client recovery [ 8040.757388] loop2: detected capacity change from 0 to 204800000 [ 8040.977693] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8041.015497] Lustre: 260795:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8041.022424] Lustre: 260795:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8041.029130] Lustre: 260795:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8041.035772] Lustre: 260795:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8041.042740] Lustre: 260795:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8041.049507] Lustre: 260795:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8041.056346] Lustre: 260795:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8041.180677] Lustre: Found index 0 for lustre-OST0000, updating log [ 8041.215815] Lustre: 260814:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8041.272664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8041.287514] Lustre: lustre-OST0000: Aborting recovery [ 8042.344721] loop3: detected capacity change from 0 to 204800000 [ 8042.541903] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8042.630377] Lustre: 260879:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8042.656838] Lustre: 260879:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8042.675092] Lustre: 260879:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8042.687411] Lustre: 260879:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8042.700404] Lustre: 260879:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8042.712741] Lustre: 260879:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8042.733172] Lustre: 260879:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8042.764327] Lustre: Found index 1 for lustre-OST0001, updating log [ 8042.811405] Lustre: 260884:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8042.907831] Lustre: lustre-OST0001: Aborting recovery [ 8048.039164] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8048.048062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8048.101173] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 8048.109735] LustreError: 260659:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 8048.123560] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 8050.111200] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 8050.113412] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 8051.428469] LustreError: 260626:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 6800us: rc = -2 [ 8051.930148] LustreError: 260624:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 720us: rc = -2 [ 8051.960292] LustreError: 260624:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 21 previous similar messages [ 8065.509853] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8065.510651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8065.511623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8065.528064] Lustre: Skipped 2 previous similar messages [ 8070.626452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8070.636732] Lustre: Skipped 6 previous similar messages [ 8071.011692] Lustre: server umount lustre-MDT0000 complete [ 8071.923231] LustreError: 260610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860990 with bad export cookie 193247208714396524 [ 8071.924968] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8071.940342] LustreError: 260610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8075.745042] LustreError: 261115:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8075.745900] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8075.746228] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8075.776443] LustreError: 261115:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 8075.795241] Lustre: Skipped 1 previous similar message [ 8076.771081] LustreError: 261115:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8076.803151] LustreError: 261115:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 8078.449804] Lustre: server umount lustre-MDT0001 complete [ 8085.552644] Lustre: server umount lustre-OST0000 complete [ 8092.936984] Lustre: server umount lustre-OST0001 complete [ 8102.576649] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 16:03:38 (1787861018) [ 8132.443395] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 8143.529454] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 8185.349707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8185.678154] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8185.734195] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8185.787893] Lustre: lustre-MDT0000: new disk, initializing [ 8185.834888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8185.842554] Lustre: Skipped 1 previous similar message [ 8185.856686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8190.425340] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8202.504699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8202.654814] Lustre: 264953:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8202.700395] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8202.703977] Lustre: Skipped 1 previous similar message [ 8202.829627] Lustre: lustre-MDT0001: new disk, initializing [ 8203.091383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8203.199781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8203.219721] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8209.152263] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8219.478682] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8226.101444] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8232.809195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8233.071930] Lustre: lustre-OST0000: new disk, initializing [ 8233.079656] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8233.088496] Lustre: 266923:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8233.183338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8238.649286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8238.666881] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8238.747684] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8239.223208] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8248.501299] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8253.784545] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8262.594829] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8270.283205] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 8276.287684] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 8321.001794] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8321.004457] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8321.016741] LustreError: Skipped 1 previous similar message [ 8321.024368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8321.036914] Lustre: Skipped 1 previous similar message [ 8321.053605] Lustre: Skipped 4 previous similar messages [ 8324.503212] Lustre: server umount lustre-OST0000 complete [ 8331.238095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8331.238280] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8331.259352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8331.269527] Lustre: Skipped 1 previous similar message [ 8334.087808] Lustre: server umount lustre-MDT0000 complete [ 8336.355477] LustreError: 264964:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8336.369264] LustreError: 264964:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 8337.776133] LustreError: 264944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861256 with bad export cookie 193247208714399226 [ 8337.787381] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8337.789146] LustreError: 264944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8338.062791] Lustre: server umount lustre-MDT0001 complete [ 8347.839805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8353.453873] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8359.466286] Lustre: server umount MGS complete [ 8371.630342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8372.180616] LustreError: 270790:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8372.207531] LustreError: 270803:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 8372.306301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8376.953209] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8377.318316] LustreError: 270791:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8386.894714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8391.857401] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8400.178823] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8406.539795] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8413.706951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8414.121449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8414.127748] Lustre: Skipped 1 previous similar message [ 8420.850461] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8429.515371] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8462.306048] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8462.312063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8462.324542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8462.333136] Lustre: Skipped 1 previous similar message [ 8462.361117] Lustre: Skipped 2 previous similar messages [ 8465.790240] Lustre: server umount lustre-OST0000 complete [ 8469.477335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8472.547960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8472.569458] Lustre: Skipped 1 previous similar message [ 8476.104462] Lustre: server umount lustre-MDT0000 complete [ 8477.666873] LustreError: 271520:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8477.710834] LustreError: 271520:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 8480.543341] LustreError: 272635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861399 with bad export cookie 193247208714401088 [ 8480.543787] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8480.554789] LustreError: 272635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8480.918782] Lustre: server umount lustre-MDT0001 complete [ 8493.024342] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 8496.247511] Key type lgssc unregistered [ 8496.618675] LNet: 274738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8496.628695] LNetError: 274738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8496.659288] LNet: Removed LNI 192.168.204.123@tcp [ 8497.487271] Key type .llcrypt unregistered [ 8497.489667] Key type ._llcrypt unregistered [ 8543.361145] Key type ._llcrypt registered [ 8543.363633] Key type .llcrypt registered [ 8543.529411] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 8562.295373] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 8563.485136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8563.565121] alg: No test for adler32 (adler32-zlib) [ 8564.657830] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 8564.922956] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 8566.656193] Key type lgssc registered [ 8568.091861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8607.071423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8607.095943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8608.483414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8608.563851] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8608.721624] Lustre: lustre-MDT0000: new disk, initializing [ 8608.854400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8608.887084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8615.118697] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8630.826352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8631.055738] Lustre: 278699:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8631.154749] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8631.160808] Lustre: Skipped 1 previous similar message [ 8631.277279] Lustre: lustre-MDT0001: new disk, initializing [ 8631.572966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8631.645769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8631.676267] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8636.769623] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8649.122371] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8655.588756] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8665.143293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8665.499185] Lustre: lustre-OST0000: new disk, initializing [ 8665.503443] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8665.518225] Lustre: 280671:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8665.641099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8672.624244] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8673.389937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8673.403184] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8673.514742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8683.276369] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8691.265908] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8701.429784] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8705.994736] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 8760.810967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8760.811717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8760.825490] LustreError: Skipped 1 previous similar message [ 8760.838719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8760.844497] Lustre: Skipped 1 previous similar message [ 8763.447281] Lustre: server umount lustre-OST0000 complete [ 8769.505487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8769.520440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8769.537534] Lustre: Skipped 1 previous similar message [ 8769.547366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8771.041583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8771.043159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8776.163601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8776.177738] Lustre: Skipped 2 previous similar messages [ 8781.284853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8781.292029] Lustre: Skipped 1 previous similar message [ 8782.307617] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8782.463093] Lustre: server umount lustre-MDT0000 complete [ 8786.612029] LustreError: 282024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861705 with bad export cookie 14970469669178471315 [ 8786.621152] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8786.625826] LustreError: 282024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8787.021734] Lustre: server umount lustre-MDT0001 complete [ 8795.468686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8800.398146] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8805.737583] Lustre: server umount MGS complete [ 8820.107172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8821.085225] LustreError: 284448:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8821.108023] LustreError: 284448:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 8821.133964] LustreError: 284461:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 8821.202259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8826.155805] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8826.342691] LustreError: 284449:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8831.470088] LustreError: 284448:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8836.586374] LustreError: 284449:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8837.527125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8838.175028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8843.018800] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8853.094735] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8861.084081] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8869.298624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8869.962497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8877.623788] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 8886.524639] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8928.228762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8928.230246] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8928.239241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8928.260202] Lustre: Skipped 1 previous similar message [ 8934.160873] Lustre: server umount lustre-OST0000 complete [ 8940.520115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8940.539744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8945.581855] Lustre: server umount lustre-MDT0000 complete [ 8948.706386] LustreError: 284449:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8948.734222] LustreError: 284449:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 8950.532817] LustreError: 284431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861869 with bad export cookie 14970469669178473254 [ 8950.542263] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8950.574541] LustreError: 284431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8951.367651] Lustre: server umount lustre-MDT0001 complete [ 8964.507762] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 8967.075880] Key type lgssc unregistered [ 8967.339242] LNet: 288418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8967.353062] LNetError: 288418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8967.375423] LNet: Removed LNI 192.168.204.123@tcp [ 8968.229467] Key type .llcrypt unregistered [ 8968.231675] Key type ._llcrypt unregistered [ 8990.520933] Key type ._llcrypt registered [ 8990.527513] Key type .llcrypt registered [ 8990.656554] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 9005.048879] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 9006.538884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9006.778830] alg: No test for adler32 (adler32-zlib) [ 9007.805584] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 9008.043373] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 9009.704807] Key type lgssc registered [ 9010.638379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9037.355314] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 9063.445356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9063.497278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9064.905854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9064.963572] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9065.162887] Lustre: lustre-MDT0000: new disk, initializing [ 9065.343095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9065.370046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9071.447981] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [ 9083.853757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9084.153182] Lustre: lustre-OST0000: new disk, initializing [ 9084.160218] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9084.169306] Lustre: Skipped 1 previous similar message [ 9084.180620] Lustre: 292473:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9084.266596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9090.674067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9090.693138] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9090.751244] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9090.834655] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10109.414726] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10109.421358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10112.818651] Lustre: server umount lustre-MDT0000 complete [10117.638070] LustreError: 293481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863036 with bad export cookie 5790818355270444402 [10117.643902] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10117.650404] LustreError: 293481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10117.785391] Lustre: server umount lustre-OST0000 complete [10150.566703] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [10162.867422] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [10209.900462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10210.414354] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10210.472362] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10210.605852] Lustre: lustre-MDT0000: new disk, initializing [10210.701260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10210.748048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10215.864762] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10230.280058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10230.429522] Lustre: 296840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10230.548327] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10230.558963] Lustre: Skipped 1 previous similar message [10230.683633] Lustre: lustre-MDT0001: new disk, initializing [10230.904739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10230.950526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10230.979343] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10235.730811] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10246.703810] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10254.374368] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10262.143799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10262.644097] Lustre: lustre-OST0000: new disk, initializing [10262.651655] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10262.667476] Lustre: 298810:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10262.754462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10270.671617] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10270.822827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10270.836252] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10270.999606] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10281.651322] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10287.020775] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10287.265623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10292.462728] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10292.726754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10296.804676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10296.808480] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10296.831166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10296.847781] Lustre: Skipped 1 previous similar message [10300.832672] Lustre: server umount lustre-OST0000 complete [10307.044324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10307.051631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10307.064538] Lustre: Skipped 1 previous similar message [10307.071204] Lustre: Skipped 1 previous similar message [10311.881550] Lustre: server umount lustre-MDT0000 complete [10315.853357] LustreError: 296833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863234 with bad export cookie 5790818355273385221 [10315.859187] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10315.868677] LustreError: 296833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10316.266353] Lustre: server umount lustre-MDT0001 complete [10320.227821] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [10322.279930] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 16:40:38 (1787863238) [10330.028641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10330.597230] LustreError: 301124:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10330.616784] LustreError: 301124:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [10330.700055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10335.565807] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10335.719493] LustreError: 301125:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10340.839536] LustreError: 301124:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10344.481320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10344.864155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10349.600777] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10357.763869] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10363.153594] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10370.108890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10370.600918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10377.385966] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10385.842053] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10394.976674] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10395.131586] Lustre: lustre-OST0001: new disk, initializing [10395.135258] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10395.147643] Lustre: 303961:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10395.238476] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10401.899909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10401.924148] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10401.974952] LustreError: 304265:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10402.610135] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10403.049650] LustreError: 304265:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10403.068873] LustreError: 304265:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [10404.078391] LustreError: 304269:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10406.199029] LustreError: 304269:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10406.213256] LustreError: 304269:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 4 previous similar messages [10410.277109] LustreError: 304265:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10410.286987] LustreError: 304265:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 6 previous similar messages [10413.123733] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10413.381878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10418.469764] LustreError: 304265:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10418.484030] LustreError: 304265:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [10419.097587] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8dc947ea6800.ost_server_uuid 50 [10420.620976] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8dc947ea6800.ost_server_uuid in FULL state after 0 sec [10426.299520] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8dc947ea6800.ost_server_uuid 50 [10427.936664] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8dc947ea6800.ost_server_uuid in FULL state after 0 sec [10435.052808] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10444.319682] Lustre: lustre-OST0001: Not available for connect from 192.168.204.23@tcp (stopping) [10444.335735] Lustre: Skipped 1 previous similar message [10445.283929] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10445.291273] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10445.300867] LustreError: Skipped 2 previous similar messages [10448.367854] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10448.371291] Lustre: Skipped 3 previous similar messages [10458.082459] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10458.291138] Lustre: server umount lustre-OST0001 complete [10458.608598] LustreError: 303758:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10458.629817] LustreError: 303758:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10463.727125] LustreError: 302946:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10463.749722] LustreError: 302946:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [10467.301874] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10467.312266] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10467.335527] Lustre: Skipped 1 previous similar message [10467.348542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10467.359373] Lustre: Skipped 4 previous similar messages [10472.632968] Lustre: server umount lustre-OST0000 complete [10478.049102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10478.065511] LustreError: Skipped 1 previous similar message [10478.069325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10478.085505] Lustre: Skipped 1 previous similar message [10484.193364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10484.209510] Lustre: Skipped 7 previous similar messages [10491.360122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10491.714837] Lustre: server umount lustre-MDT0000 complete [10494.438667] LustreError: 301124:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10494.460972] LustreError: 301124:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [10496.229984] LustreError: 301105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863414 with bad export cookie 5790818355273386103 [10496.235832] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10496.243512] LustreError: 301105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10496.627895] Lustre: server umount lustre-MDT0001 complete [10508.680428] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [10511.349412] Key type lgssc unregistered [10511.660229] LNet: 306654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10511.677137] LNetError: 306654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10511.697700] LNet: Removed LNI 192.168.204.123@tcp [10512.582883] Key type .llcrypt unregistered [10512.590549] Key type ._llcrypt unregistered [10537.547837] Key type ._llcrypt registered [10537.554025] Key type .llcrypt registered [10537.706800] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [10538.985042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10538.998468] alg: No test for adler32 (adler32-zlib) [10540.043388] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [10540.301435] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [10542.000233] Key type lgssc registered [10543.289555] Lustre: Echo OBD driver; http://www.lustre.org/ [10552.518605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10552.550402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10554.263388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10560.685711] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10563.557259] LustreError: 308076:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10563.571834] LustreError: 308076:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [10568.678898] LustreError: 308077:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10571.146126] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10573.793176] LustreError: 308076:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10578.926961] LustreError: 308077:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10579.247657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10579.663857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10585.002133] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10594.831118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10595.213794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10599.355186] LustreError: 309759:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10599.399088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [10601.808031] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10609.640630] LustreError: 309758:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10609.663870] LustreError: 309758:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [10609.751794] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10616.842597] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10617.170450] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10622.455987] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [10623.352749] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10630.548649] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10637.992410] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [10638.191800] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10743.273061] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10743.288782] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10743.293298] Lustre: Skipped 1 previous similar message [10748.395180] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10748.403741] Lustre: Skipped 1 previous similar message [10749.428303] Lustre: server umount lustre-OST0001 complete [10753.528482] LustreError: 311051:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10753.540582] LustreError: 311051:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [10758.124410] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10758.129847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10758.142606] Lustre: Skipped 1 previous similar message [10758.147399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10758.151552] Lustre: Skipped 1 previous similar message [10761.981146] Lustre: server umount lustre-OST0000 complete [10768.865053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10768.866913] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10768.882699] LustreError: Skipped 1 previous similar message [10768.908207] Lustre: Skipped 1 previous similar message [10768.908950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10771.842695] Lustre: server umount lustre-MDT0000 complete [10776.133738] LustreError: 308056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863694 with bad export cookie 8148891765455671383 [10776.136765] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10776.146756] LustreError: 308056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10776.462098] Lustre: server umount lustre-MDT0001 complete [10788.395157] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [10791.382954] Key type lgssc unregistered [10791.671210] LNet: 313027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10791.680839] LNetError: 313027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10791.710646] LNet: Removed LNI 192.168.204.123@tcp [10792.582952] Key type .llcrypt unregistered [10792.590805] Key type ._llcrypt unregistered [10840.076575] Key type ._llcrypt registered [10840.079477] Key type .llcrypt registered [10840.187307] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [10841.027553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10841.174332] alg: No test for adler32 (adler32-zlib) [10842.182532] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [10842.395485] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [10844.064202] Key type lgssc registered [10845.118921] Lustre: Echo OBD driver; http://www.lustre.org/ [10854.979228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10855.016672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10856.589118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10861.813612] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10867.179613] LustreError: 315318:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10867.223297] LustreError: 315318:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [10870.352172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10870.716473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10874.367623] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10876.857757] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10883.213310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10883.586261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10889.992642] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10895.859880] LustreError: 316809:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10899.228379] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10899.665963] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10901.701703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [10901.713871] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [10907.290578] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [10916.622775] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10924.619906] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10942.442636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10942.450698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10942.465636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10942.972255] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10942.986580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10943.001713] Lustre: Skipped 1 previous similar message [10943.030033] Lustre: Skipped 2 previous similar messages [10944.885098] Lustre: server umount lustre-MDT0000 complete [10948.079191] LustreError: 316408:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10948.102048] LustreError: 316408:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [10949.325796] LustreError: 318326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863867 with bad export cookie 15639522092128446466 [10949.341455] LustreError: 318326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10949.342139] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10949.689482] Lustre: server umount lustre-MDT0001 complete [10965.318649] Lustre: server umount lustre-OST0000 complete [10968.544093] Lustre: 314579:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787863871/real 1787863871] req@ffff94b5c3598b40 x1874711030071168/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787863887 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10968.600768] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10968.616700] Lustre: Skipped 1 previous similar message [10969.568263] Lustre: 314582:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787863872/real 1787863872] req@ffff94b5cecef480 x1874711030071424/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787863888 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10970.254573] Lustre: server umount lustre-OST0001 complete [10990.618087] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [11002.088809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11002.867749] LustreError: 320581:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11003.003366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11008.487356] LustreError: 320582:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11009.226574] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11017.718347] LustreError: 320581:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11017.749993] LustreError: 320581:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [11019.127053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11020.090621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11025.891708] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11029.564159] Lustre: 321722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11038.502253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11038.928727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11045.051619] LustreError: 322077:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11047.840056] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11049.239381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [11058.819670] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11064.331305] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [11066.999757] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11076.361985] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11080.550252] Lustre: 323598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11086.819575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11086.838743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11086.857547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11089.896160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11091.253882] Lustre: server umount lustre-MDT0000 complete [11095.018383] LustreError: 321533:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11095.041547] LustreError: 321533:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 7 previous similar messages [11095.908543] LustreError: 320562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864014 with bad export cookie 15639522092128448167 [11095.912547] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11095.916754] LustreError: 320562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11096.494286] Lustre: server umount lustre-MDT0001 complete [11111.783729] Lustre: server umount lustre-OST0000 complete [11125.480926] Lustre: server umount lustre-OST0001 complete [11128.672679] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [11129.933116] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [11131.347299] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 16:54:08 (1787864048) [11156.515255] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [11166.371340] loop0: detected capacity change from 0 to 1099511627776 [11167.108502] SGI XFS with ACLs, security attributes, quota, no debug enabled [11167.116763] *** VALIDATE xfs *** [11167.131711] XFS (loop0): Mounting V5 Filesystem [11167.162351] XFS (loop0): Ending clean mount [11168.597572] loop1: detected capacity change from 0 to 18691697672192 [11275.182177] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [11289.450189] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [11325.492385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11325.665164] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11325.685723] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11325.766209] Lustre: lustre-MDT0000: new disk, initializing [11325.840531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11325.853150] Lustre: Skipped 1 previous similar message [11325.881703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11330.720634] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11341.363505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11341.509843] Lustre: 328756:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11341.580325] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11341.584339] Lustre: Skipped 1 previous similar message [11341.735295] Lustre: lustre-MDT0001: new disk, initializing [11341.905739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11341.921392] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11346.535552] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11354.942408] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11359.900350] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11365.738861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11365.956580] Lustre: lustre-OST0000: new disk, initializing [11365.963748] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11365.968889] Lustre: 330726:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11367.596464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11367.604199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11367.730449] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11371.278325] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11380.764470] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11385.759394] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11385.957523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11390.309786] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11390.477305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11393.506208] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11393.509316] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11393.513942] LustreError: Skipped 1 previous similar message [11393.515155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11393.532204] Lustre: Skipped 4 previous similar messages [11393.551612] Lustre: Skipped 4 previous similar messages [11403.748731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11403.756546] Lustre: Skipped 3 previous similar messages [11406.304385] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11406.465829] Lustre: server umount lustre-OST0000 complete [11413.476589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11413.490466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11415.728194] Lustre: server umount lustre-MDT0000 complete [11419.076106] LustreError: 329562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864337 with bad export cookie 15639522092128449714 [11419.085334] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11419.089019] LustreError: 329562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11419.372363] Lustre: server umount lustre-MDT0001 complete [11426.372483] XFS (loop0): Unmounting Filesystem [11430.399765] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 16:59:07 (1787864347) [11435.680399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11436.135092] LustreError: 333283:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11436.187773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11436.195447] Lustre: Skipped 2 previous similar messages [11440.085245] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11448.379463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11453.158533] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11460.060404] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11465.987797] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11472.422989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11478.701627] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11485.726394] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11494.369368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11494.375625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11494.401112] Lustre: Skipped 1 previous similar message [11494.410635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11494.417178] Lustre: Skipped 2 previous similar messages [11499.120993] Lustre: server umount lustre-OST0000 complete [11505.122200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11505.135879] LustreError: Skipped 1 previous similar message [11508.778927] Lustre: server umount lustre-MDT0000 complete [11512.623848] LustreError: 335126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864431 with bad export cookie 15639522092128450603 [11512.625157] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11512.633872] LustreError: 335126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11513.037907] Lustre: server umount lustre-MDT0001 complete [11523.476279] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [11526.495435] Key type lgssc unregistered [11526.799137] LNet: 336915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11526.811813] LNetError: 336915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11526.824331] LNet: Removed LNI 192.168.204.123@tcp [11527.675181] Key type .llcrypt unregistered [11527.682186] Key type ._llcrypt unregistered [11555.871339] Key type ._llcrypt registered [11555.874488] Key type .llcrypt registered [11556.005889] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [11557.114462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11557.206611] alg: No test for adler32 (adler32-zlib) [11558.275623] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [11558.558434] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [11560.240156] Key type lgssc registered [11561.197965] Lustre: Echo OBD driver; http://www.lustre.org/ [11568.575412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11568.612939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11570.151777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11574.353406] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11580.389259] LustreError: 338495:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11580.405092] LustreError: 338495:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11582.842387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11583.214541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11587.523666] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11594.868309] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11600.439364] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11607.030964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11607.276805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11613.545694] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11621.594344] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11628.540383] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.23@tcp (stopping) [11629.540504] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11629.549611] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11629.563151] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11631.084291] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11631.092449] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11633.631552] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.23@tcp (stopping) [11633.646192] Lustre: Skipped 1 previous similar message [11634.913147] Lustre: server umount lustre-MDT0001 complete [11636.200834] LustreError: 338491:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11636.213848] LustreError: 338491:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [11638.750759] LustreError: 338490:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11641.314565] LustreError: 338491:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11641.344371] LustreError: 338491:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [11646.439626] LustreError: 338495:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11646.462568] LustreError: 338495:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11646.598609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11646.668217] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11646.672158] Lustre: Found index 1 for lustre-MDT0001, updating log [11646.720666] Lustre: 338482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11647.055844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11651.520504] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11652.078877] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11652.104827] LustreError: 341969:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11652.118305] LustreError: 341969:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11652.145696] LustreError: 338487:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11652.243677] LustreError: 341976:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.123@tcp: cfg command failed: rc = -17 [11652.265752] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [11652.265752] [11662.319988] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11662.329840] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11662.365146] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11662.385311] Lustre: Skipped 1 previous similar message [11667.904436] Lustre: server umount lustre-MDT0001 complete [11669.475750] LustreError: 339733:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11680.611882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11680.727523] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11680.736042] Lustre: Found index 1 for lustre-MDT0001, updating log [11680.794912] Lustre: 338485:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11681.366365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11686.383305] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11686.855677] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [11688.968681] LustreError: 343126:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11688.977685] LustreError: 343126:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11688.988177] LustreError: 338487:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11689.001497] LustreError: 338487:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [11689.064097] LustreError: 343134:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [11689.069032] LustreError: 343134:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [11689.073739] LustreError: 343134:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.123@tcp: cfg command failed: rc = -17 [11689.081833] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.204.123@tcp [11689.081833] [11992.061329] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.23@tcp (stopping) [11992.068833] Lustre: Skipped 3 previous similar messages [11993.571679] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11993.596773] Lustre: Skipped 1 previous similar message [11997.573354] Lustre: server umount lustre-MDT0001 complete [11998.693135] LustreError: 338495:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11998.713325] LustreError: 338495:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 7 previous similar messages [12007.225929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12007.352248] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12007.357122] Lustre: Found index 1 for lustre-MDT0001, updating log [12007.399805] Lustre: 338485:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12007.704444] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12012.364355] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [12013.054903] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12017.148034] LustreError: 351558:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12017.156558] LustreError: 351558:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12017.171913] LustreError: 338487:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12017.179966] LustreError: 338487:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12017.228964] LustreError: 351561:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [12017.247332] LustreError: 351561:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.123@tcp: cfg command failed: rc = -114 [12017.253631] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [12017.253631] [12335.596233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12335.608189] Lustre: Skipped 1 previous similar message [12335.616248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12335.623513] Lustre: Skipped 5 previous similar messages [12348.385969] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12348.733390] Lustre: server umount lustre-MDT0000 complete [12350.948298] LustreError: 341656:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12350.968537] LustreError: 341656:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [12352.958886] LustreError: 338520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865271 with bad export cookie 7614436371322898222 [12352.961182] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12352.982225] LustreError: 338520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12356.078072] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12356.101768] Lustre: Skipped 2 previous similar messages [12367.328541] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12367.688186] Lustre: server umount lustre-MDT0001 complete [12372.861610] Lustre: server umount lustre-OST0000 complete [12384.005922] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [12394.884358] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [12430.290424] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 17:15:46 (1787865346) [12436.548747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12436.743882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12436.775140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12436.878098] Lustre: lustre-MDT0000: new disk, initializing [12436.967125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12436.980930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12441.180170] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [12451.053921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12451.188802] Lustre: 362331:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12451.267661] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12451.273403] Lustre: Skipped 1 previous similar message [12451.399621] Lustre: lustre-MDT0001: new disk, initializing [12451.597893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12451.654229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12451.668070] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12455.746728] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [12465.038996] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12472.100923] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12479.861251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12480.260771] Lustre: lustre-OST0000: new disk, initializing [12480.265954] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12480.283130] Lustre: 364299:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12480.399391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12487.194788] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all [12487.232940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12487.247335] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12487.384215] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12498.080949] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12507.110313] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12507.131526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12507.155683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12507.164717] Lustre: Skipped 10 previous similar messages [12508.139878] LustreError: 362337:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12508.153137] LustreError: 362337:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [12508.193787] Lustre: server umount lustre-MDT0000 complete [12511.698768] LustreError: 362323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865430 with bad export cookie 7614436371322900168 [12511.704839] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12511.712812] LustreError: 362323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12511.731949] LustreError: 363138:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [12512.021262] Lustre: server umount lustre-MDT0001 complete [12526.184763] Lustre: server umount lustre-OST0000 complete [12552.492724] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [12562.035711] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [12593.232775] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12236 sec ========= 17:18:29 (1787865509) [12595.565742] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 17:18:31 (1787865511) === [12599.094528] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 17:18:35 (1787865515) === [12640.347440] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [12643.356464] Key type lgssc unregistered [12643.701499] LNet: 370666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12643.712497] LNetError: 370666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12643.748092] LNet: Removed LNI 192.168.204.123@tcp [12644.990200] Key type .llcrypt unregistered [12644.991844] Key type ._llcrypt unregistered