[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 643923081 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003040] x2apic enabled [ 0.005003] Switched APIC routing to physical x2apic. [ 0.006010] kvm-guest: setup PV IPIs [ 0.008755] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010010] pid_max: default: 32768 minimum: 301 [ 0.012038] LSM: Security Framework initializing [ 0.013039] Yama: becoming mindful. [ 0.014030] SELinux: Initializing. [ 0.015054] *** VALIDATE selinux *** [ 0.023680] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028136] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029292] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030176] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031116] *** VALIDATE tmpfs *** [ 0.032525] *** VALIDATE proc *** [ 0.034167] *** VALIDATE cgroup *** [ 0.035008] *** VALIDATE cgroup2 *** [ 0.036244] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037140] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039030] Spectre V2 : User space: Vulnerable [ 0.040007] Speculative Store Bypass: Vulnerable [ 0.043335] debug: unmapping init [mem 0xffffffffab059000-0xffffffffab060fff] [ 0.045282] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046929] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048038] ... version: 2 [ 0.049008] ... bit width: 48 [ 0.050008] ... generic registers: 4 [ 0.051006] ... value mask: 0000ffffffffffff [ 0.052009] ... max period: 00007fffffffffff [ 0.053006] ... fixed-purpose events: 3 [ 0.054006] ... event mask: 000000070000000f [ 0.055247] rcu: Hierarchical SRCU implementation. [ 0.057521] smp: Bringing up secondary CPUs ... [ 0.058578] x86: Booting SMP configuration: [ 0.059015] .... node #0, CPUs: #1 #2 #3 [ 0.071144] smp: Brought up 1 node, 4 CPUs [ 0.073018] smpboot: Max logical packages: 1 [ 0.074009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.115044] node 0 deferred pages initialised in 39ms [ 0.118159] devtmpfs: initialized [ 0.119198] x86/mm: Memory block size: 128MB [ 0.122010] gcov: version magic: 0x41383552 [ 0.124030] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.126214] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.129307] pinctrl core: initialized pinctrl subsystem [ 0.131137] [ 0.131776] ************************************************************* [ 0.134010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.136008] ** ** [ 0.138011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.140009] ** ** [ 0.142010] ** This means that this kernel is built to expose internal ** [ 0.145014] ** IOMMU data structures, which may compromise security on ** [ 0.147008] ** your system. ** [ 0.149010] ** ** [ 0.151010] ** If you see this message and you are not debugging the ** [ 0.153008] ** kernel, report this immediately to your vendor! ** [ 0.155010] ** ** [ 0.157010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159012] ************************************************************* [ 0.161930] NET: Registered protocol family 16 [ 0.164436] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.166112] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.169061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.173773] cpuidle: using governor menu [ 0.175936] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.178606] PCI: Using configuration type 1 for base access [ 0.181155] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.191061] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.192015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.195073] cryptd: max_cpu_qlen set to 1000 [ 0.196197] ACPI: Added _OSI(Module Device) [ 0.197013] ACPI: Added _OSI(Processor Device) [ 0.198041] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.199012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.205738] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.215341] ACPI: Interpreter enabled [ 0.216051] ACPI: PM: (supports S0 S3 S4 S5) [ 0.217007] ACPI: Using IOAPIC for interrupt routing [ 0.218103] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.222424] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.233536] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.237047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.240022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.244077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.248393] acpiphp: Slot [2] registered [ 0.250126] acpiphp: Slot [5] registered [ 0.251071] acpiphp: Slot [6] registered [ 0.253101] acpiphp: Slot [7] registered [ 0.254086] acpiphp: Slot [8] registered [ 0.255076] acpiphp: Slot [9] registered [ 0.256082] acpiphp: Slot [10] registered [ 0.258100] acpiphp: Slot [3] registered [ 0.259058] acpiphp: Slot [4] registered [ 0.260093] acpiphp: Slot [11] registered [ 0.261074] acpiphp: Slot [12] registered [ 0.263066] acpiphp: Slot [13] registered [ 0.264069] acpiphp: Slot [14] registered [ 0.265080] acpiphp: Slot [15] registered [ 0.266073] acpiphp: Slot [16] registered [ 0.268088] acpiphp: Slot [17] registered [ 0.269084] acpiphp: Slot [18] registered [ 0.270076] acpiphp: Slot [19] registered [ 0.271079] acpiphp: Slot [20] registered [ 0.272064] acpiphp: Slot [21] registered [ 0.274113] acpiphp: Slot [22] registered [ 0.275014] acpiphp: Slot [23] registered [ 0.276111] acpiphp: Slot [24] registered [ 0.277071] acpiphp: Slot [25] registered [ 0.278029] acpiphp: Slot [26] registered [ 0.279025] acpiphp: Slot [27] registered [ 0.279979] acpiphp: Slot [28] registered [ 0.281146] acpiphp: Slot [29] registered [ 0.283093] acpiphp: Slot [30] registered [ 0.284000] acpiphp: Slot [31] registered [ 0.284000] PCI host bridge to bus 0000:00 [ 0.284000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.284000] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.284000] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.284000] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.284000] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.284000] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.284000] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.284000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.284000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.284000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.284000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.284000] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.284000] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.284000] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.284000] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.284000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.284000] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.284000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.284000] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.284000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.285074] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.294513] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.319014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.327014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.361016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.395923] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.585107] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.622025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.651018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.673000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.693027] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.718021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.830051] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.854710] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.865000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.876016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.900035] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.913622] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.922016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.928021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.944062] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.955822] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.966039] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.979066] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.008132] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.024703] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.027573] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.030669] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.033584] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.036326] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.043130] iommu: Default domain type: Passthrough [ 1.047135] SCSI subsystem initialized [ 1.049158] ACPI: bus type USB registered [ 1.051223] usbcore: registered new interface driver usbfs [ 1.055116] usbcore: registered new interface driver hub [ 1.057095] usbcore: registered new device driver usb [ 1.059154] pps_core: LinuxPPS API ver. 1 registered [ 1.061011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.066126] PTP clock support registered [ 1.069034] EDAC MC: Ver: 3.0.0 [ 1.070318] PCI: Using ACPI for IRQ routing [ 1.071000] NetLabel: Initializing [ 1.072009] NetLabel: domain hash size = 128 [ 1.075030] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.079182] NetLabel: unlabeled traffic allowed by default [ 1.084625] vgaarb: loaded [ 1.087568] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.090011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.100257] clocksource: Switched to clocksource kvm-clock [ 1.279957] VFS: Disk quotas dquot_6.6.0 [ 1.285628] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.291748] *** VALIDATE ramfs *** [ 1.294491] *** VALIDATE hugetlbfs *** [ 1.297237] pnp: PnP ACPI init [ 1.300783] pnp: PnP ACPI: found 6 devices [ 1.326603] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.331822] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.334906] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.337842] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.341134] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.344997] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.349617] NET: Registered protocol family 2 [ 1.354139] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.360456] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.364972] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.371052] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.375200] TCP: Hash tables configured (established 65536 bind 65536) [ 1.378178] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.380641] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.383344] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.386769] NET: Registered protocol family 1 [ 1.388819] RPC: Registered named UNIX socket transport module. [ 1.391729] RPC: Registered udp transport module. [ 1.393796] RPC: Registered tcp transport module. [ 1.395493] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.397933] NET: Registered protocol family 44 [ 1.399776] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.401996] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.404246] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.406540] PCI: CLS 0 bytes, default 64 [ 1.408347] Unpacking initramfs... [ 3.735790] debug: unmapping init [mem 0xffff9c8cfcc54000-0xffff9c8cfffbffff] [ 3.740925] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.744162] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.748101] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.323818] Initialise system trusted keyrings [ 4.325663] Key type blacklist registered [ 4.327343] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.335099] zbud: loaded [ 4.337443] *** VALIDATE nfs *** [ 4.339190] *** VALIDATE nfs4 *** [ 4.340812] pstore: using deflate compression [ 4.343746] Platform Keyring initialized [ 4.450389] NET: Registered protocol family 38 [ 4.452746] Key type asymmetric registered [ 4.454520] Asymmetric key parser 'x509' registered [ 4.456932] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.461952] io scheduler mq-deadline registered [ 4.488163] io scheduler kyber registered [ 4.491320] io scheduler bfq registered [ 4.493279] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.498497] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.503776] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.508948] ACPI: Power Button [PWRF] [ 4.515874] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.527083] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.562503] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.579988] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.666353] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.697747] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.729828] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.736130] Non-volatile memory driver v1.3 [ 4.738552] Linux agpgart interface v0.103 [ 4.770212] virtio_blk virtio1: [vda] 149952 512-byte logical blocks (76.8 MB/73.2 MiB) [ 4.774303] vda: detected capacity change from 0 to 76775424 [ 4.799443] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.806185] vdb: detected capacity change from 0 to 1073741824 [ 4.826944] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.832770] vdc: detected capacity change from 0 to 2621440000 [ 4.863959] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.870631] vdd: detected capacity change from 0 to 2621440000 [ 4.899836] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.907948] vde: detected capacity change from 0 to 4294967296 [ 4.929949] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.936292] vdf: detected capacity change from 0 to 4294967296 [ 4.952147] libphy: Fixed MDIO Bus: probed [ 4.965740] usbcore: registered new interface driver usbserial_generic [ 4.970534] usbserial: USB Serial support registered for generic [ 4.976290] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.984366] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.987749] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.992835] mousedev: PS/2 mouse device common for all mice [ 4.997728] rtc_cmos 00:05: RTC can wake from S4 [ 5.002673] rtc_cmos 00:05: registered as rtc0 [ 5.005826] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.010332] intel_pstate: CPU model not supported [ 5.014478] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.019650] hid: raw HID events driver (C) Jiri Kosina [ 5.028742] usbcore: registered new interface driver usbhid [ 5.033096] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.034279] usbhid: USB HID core driver [ 5.034410] drop_monitor: Initializing network drop monitor service [ 5.034513] Initializing XFRM netlink socket [ 5.036057] NET: Registered protocol family 10 [ 5.050792] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.051973] Segment Routing with IPv6 [ 5.056734] NET: Registered protocol family 17 [ 5.058568] mpls_gso: MPLS GSO support [ 5.063723] RAS: Correctable Errors collector initialized. [ 5.065616] AVX version of gcm_enc/dec engaged. [ 5.067209] AES CTR mode by8 optimization enabled [ 5.152937] sched_clock: Marking stable (5152783770, 0)->(6528562149, -1375778379) [ 5.168716] registered taskstats version 1 [ 5.171397] Loading compiled-in X.509 certificates [ 5.176690] zswap: loaded using pool lzo/zbud [ 5.325458] Key type big_key registered [ 5.357546] Key type encrypted registered [ 5.362941] ima: No TPM chip found, activating TPM-bypass! [ 5.369616] ima: Allocated hash algorithm: sha1 [ 5.378218] ima: No architecture policies found [ 5.382606] evm: Initialising EVM extended attributes: [ 5.387194] evm: security.selinux [ 5.391254] evm: security.ima [ 5.394511] evm: security.capability [ 5.397484] evm: HMAC attrs: 0x1 [ 5.403820] rtc_cmos 00:05: setting system clock to 2026-09-14 06:15:53 UTC (1789366553) [ 5.415878] debug: unmapping init [mem 0xffffffffac003000-0xffffffffac1fffff] [ 5.421836] debug: unmapping init [mem 0xffffffffaad82000-0xffffffffab058fff] [ 5.433172] Write protecting the kernel read-only data: 28672k [ 5.439815] debug: unmapping init [mem 0xffffffffa9403000-0xffffffffa95fffff] [ 5.445039] debug: unmapping init [mem 0xffffffffa9d14000-0xffffffffa9dfffff] [ 5.627242] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.637866] systemd[1]: Detected virtualization kvm. [ 5.640538] systemd[1]: Detected architecture x86-64. [ 5.643575] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.685919] systemd[1]: No hostname configured. [ 5.690354] systemd[1]: Set hostname to . [ 5.694952] random: systemd: uninitialized urandom read (16 bytes read) [ 5.700235] systemd[1]: Initializing machine ID from random generator. [ 6.017772] random: systemd: uninitialized urandom read (16 bytes read) [ 6.023302] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 6.135662] random: systemd: uninitialized urandom read (16 bytes read) [ 6.140306] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 6.162296] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.964346] device-mapper: uevent: version 1.0.3 [ 7.966329] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 10.275101] random: fast init done Starting dracut initqueue hook... [ 10.329253] virtio_net virtio0 ens2: renamed from eth0 [ 11.524201] scsi host0: ata_piix [ 11.541213] scsi host1: ata_piix [ 11.542796] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.545097] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.951715] random: crng init done [ 15.978486] random: 7 urandom warning(s) missed due to ratelimiting [ 18.436321] dracut-initqueue[578]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 20.383874] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.315594] printk: systemd: 22 output lines suppressed due to ratelimiting [ 23.895258] SELinux: Disabled at runtime. [ 23.983732] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.000673] systemd[1]: Detected virtualization kvm. [ 24.003938] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.630859] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.650928] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.676051] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.693986] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.702966] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.740917] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.768329] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 25.906242] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Mounting Huge Pages File System... Starting Apply Kernel Variables... Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 27.256614] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 28.197761] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.320971] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.686252] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.833614] EDAC sbridge: Ver: 1.1.2 [ 32.839801] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 33.727527] NFS: Registering the id_resolver key type [ 33.731944] Key type id_resolver registered [ 33.748731] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Started Login Service. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg323-server login: [ 81.937656] libcfs: loading out-of-tree module taints kernel. [ 81.963918] Key type ._llcrypt registered [ 81.966514] Key type .llcrypt registered [ 82.055798] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 95.631157] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 96.693490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 96.704805] alg: No test for adler32 (adler32-zlib) [ 97.881411] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 98.507759] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 100.192089] Key type lgssc registered [ 101.337154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 107.271752] hrtimer: interrupt took 5385060 ns [ 113.149331] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 138.847791] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 147.267834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 147.284092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 148.446950] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 148.470528] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 148.532214] Lustre: lustre-MDT0000: new disk, initializing [ 148.592180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 148.613212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 151.516230] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 160.656702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 160.763847] Lustre: 6507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 160.796580] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 160.805000] Lustre: Skipped 1 previous similar message [ 160.874496] Lustre: lustre-MDT0001: new disk, initializing [ 160.926804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 160.950242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 160.957939] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 164.004596] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 167.648814] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 173.985672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 174.147125] Lustre: lustre-OST0000: new disk, initializing [ 174.150724] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 174.155690] Lustre: 8445:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 174.205436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 176.206825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 176.214993] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 176.269629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 178.113598] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.205591] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 188.299141] Lustre: lustre-OST0001: new disk, initializing [ 188.302600] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 188.307706] Lustre: 9517:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 188.363677] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 192.670704] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 198.144255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 198.156575] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 198.185082] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 201.466577] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 206.413091] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 210.588523] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing check_logdir /tmp/testlogs/ [ 213.995831] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing yml_node [ 216.996885] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 218.663296] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 220.291451] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 221.323509] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 14 02:19:28 EDT 2026 [ 231.788781] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 232.951770] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 248.293420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 248.299753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 248.313763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 249.312831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 249.313363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 249.321931] Lustre: Skipped 1 previous similar message [ 249.332648] Lustre: Skipped 3 previous similar messages [ 253.387459] Lustre: server umount lustre-MDT0000 complete [ 255.957587] LustreError: 7456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789366804 with bad export cookie 3788657235635737236 [ 255.962546] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 255.970099] LustreError: 7456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 259.551840] LustreError: 6514:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 259.552139] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 259.564940] LustreError: 6514:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 259.591822] Lustre: Skipped 1 previous similar message [ 259.596184] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 259.601692] Lustre: Skipped 1 previous similar message [ 260.576126] LustreError: 11512:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 260.594296] LustreError: 11512:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 265.702124] LustreError: 8621:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 265.709735] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 265.715806] LustreError: 8621:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 265.720328] Lustre: Skipped 3 previous similar messages [ 270.303170] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 270.497040] Lustre: server umount lustre-MDT0001 complete [ 273.319940] Lustre: server umount lustre-OST0000 complete [ 276.013553] Lustre: server umount lustre-OST0001 complete [ 281.038638] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 287.758229] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 312.565601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 312.772579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 312.807201] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 312.862499] Lustre: lustre-MDT0000: new disk, initializing [ 312.940942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 312.956659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 316.237950] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 324.373713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 324.466725] Lustre: 15058:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 324.508896] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 324.512656] Lustre: Skipped 1 previous similar message [ 324.568641] Lustre: lustre-MDT0001: new disk, initializing [ 324.661905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 324.687312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 324.711596] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 327.584322] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 334.326574] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 338.435386] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 343.395144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 343.677096] Lustre: lustre-OST0000: new disk, initializing [ 343.685382] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 343.691211] Lustre: 17028:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 343.753592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 345.408854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 345.417180] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 345.505773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 348.694917] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 356.304855] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 360.455800] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 360.652755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 364.006763] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 364.146634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 366.051321] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 366.054518] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 366.061864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 366.074949] Lustre: Skipped 2 previous similar messages [ 366.083291] Lustre: Skipped 2 previous similar messages [ 371.700446] Lustre: server umount lustre-OST0000 complete [ 375.776206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 375.786383] LustreError: Skipped 1 previous similar message [ 375.793571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 375.813210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 375.823421] Lustre: Skipped 3 previous similar messages [ 389.087121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 389.208399] Lustre: server umount lustre-MDT0000 complete [ 391.649457] LustreError: 16579:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 391.665363] LustreError: 16579:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 391.867386] LustreError: 15052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789366939 with bad export cookie 3788657235635738769 [ 391.872230] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 391.880885] LustreError: 15052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 392.122945] Lustre: server umount lustre-MDT0001 complete [ 395.947501] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 02:22:23 (1789366943) [ 400.833980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 401.155189] LustreError: 19290:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 401.208264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 404.293823] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 410.457459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 413.850087] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 419.435377] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 423.519362] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 428.390433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 433.081406] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 438.436106] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 451.985743] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 456.676127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 456.682425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 456.685199] Lustre: Skipped 2 previous similar messages [ 456.687818] Lustre: Skipped 6 previous similar messages [ 462.931263] Lustre: server umount lustre-MDT0000 complete [ 465.411977] LustreError: 19272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789367013 with bad export cookie 3788657235635739651 [ 465.412890] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 465.420140] LustreError: 19272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 465.616484] Lustre: server umount lustre-MDT0001 complete [ 478.735980] Lustre: server umount lustre-OST0000 complete [ 492.468949] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 498.510685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 498.772343] LustreError: 23687:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 498.791064] LustreError: 23687:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 498.837632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 498.842316] Lustre: Skipped 2 previous similar messages [ 501.450660] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 507.077516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 510.059551] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 512.076735] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 516.906044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 521.082551] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 526.894979] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 527.038906] Lustre: lustre-OST0001: new disk, initializing [ 527.043674] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 527.048660] Lustre: 25997:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 529.000862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 529.015658] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 529.069950] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 531.364975] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 538.348289] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 540.501319] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 548.324340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 548.331079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 548.346885] Lustre: Skipped 1 previous similar message [ 548.353967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 548.361341] Lustre: Skipped 5 previous similar messages [ 553.133636] Lustre: server umount lustre-MDT0000 complete [ 554.983378] LustreError: 23684:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 554.996598] LustreError: 23684:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 555.454911] LustreError: 23669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789367103 with bad export cookie 3788657235635740498 [ 555.456283] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 555.463996] LustreError: 23669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 555.663093] Lustre: server umount lustre-MDT0001 complete [ 568.410902] Lustre: server umount lustre-OST0000 complete [ 581.015416] Lustre: server umount lustre-OST0001 complete [ 590.984230] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 596.558889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 596.778053] LustreError: 28969:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 599.103448] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 604.350991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 607.011766] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 608.785967] Lustre: 30114:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 612.833273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 616.986054] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 622.697770] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 626.585079] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 631.339532] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 633.474407] Lustre: 31993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 638.434532] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 638.437473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 638.449370] Lustre: Skipped 5 previous similar messages [ 638.453028] Lustre: Skipped 5 previous similar messages [ 643.143082] Lustre: server umount lustre-MDT0000 complete [ 645.155826] LustreError: 28950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789367193 with bad export cookie 3788657235635741793 [ 645.158977] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 645.162682] LustreError: 28950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 645.355565] Lustre: server umount lustre-MDT0001 complete [ 657.947248] Lustre: server umount lustre-OST0000 complete [ 670.332938] Lustre: server umount lustre-OST0001 complete [ 675.382371] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 02:27:02 (1789367222) [ 691.996970] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 697.508272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 697.699846] LustreError: 34792:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 697.711705] LustreError: 34792:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 697.740230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 697.744212] Lustre: Skipped 7 previous similar messages [ 700.031332] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 704.535680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 707.245409] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 709.016218] Lustre: 35939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 713.068292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 716.988850] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 722.678364] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 726.714832] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 731.957865] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 734.319806] Lustre: 37819:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 739.301438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 739.308990] Lustre: Skipped 3 previous similar messages [ 743.424661] Lustre: server umount lustre-MDT0000 complete [ 745.380979] LustreError: 37821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789367293 with bad export cookie 3788657235635742899 [ 745.385255] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 745.388350] LustreError: 37821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 745.589128] Lustre: server umount lustre-MDT0001 complete [ 757.793483] Lustre: server umount lustre-OST0000 complete [ 770.302953] Lustre: server umount lustre-OST0001 complete [ 775.230986] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 02:28:42 (1789367322) [ 784.124916] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 789.559249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 792.017315] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 796.862513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 799.205276] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 800.863720] Lustre: 41104:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 804.797103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 808.019447] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 812.181389] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 815.420079] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 819.520893] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 821.666160] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 821.670770] Lustre: Skipped 1 previous similar message [ 824.802648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 824.806575] Lustre: Skipped 6 previous similar messages [ 829.326888] Lustre: server umount lustre-MDT0000 complete [ 829.926579] LustreError: 39957:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 829.936232] LustreError: 39957:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 831.474482] LustreError: 39937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789367379 with bad export cookie 3788657235635743998 [ 831.478304] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 831.480752] LustreError: 39937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 837.728368] Lustre: server umount lustre-MDT0001 complete [ 842.225827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 856.543784] LustreError: 3642:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9c8d4bece680 x1876286920537344/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 858.819212] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 863.301606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 865.841872] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 866.599731] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 869.710985] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 870.459703] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 873.271956] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 874.156090] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 877.028243] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 877.037517] Lustre: Skipped 9 previous similar messages [ 882.894452] Lustre: server umount lustre-MDT0000 complete [ 884.780578] LustreError: 41485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789367432 with bad export cookie 3788657235635745132 [ 884.782845] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 884.785757] LustreError: 41485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 884.948341] Lustre: server umount lustre-MDT0001 complete [ 897.543752] Lustre: server umount lustre-OST0000 complete [ 909.675523] Lustre: server umount lustre-OST0001 complete [ 914.370867] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 02:31:01 (1789367461) [ 922.460648] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 927.284614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 929.278922] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 933.249738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 935.218455] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 936.639623] Lustre: 48733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 936.647281] Lustre: 48733:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 940.011450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 943.141987] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 947.472723] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 950.352477] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 959.746858] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 961.661806] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 971.943191] Lustre: server umount lustre-MDT0000 complete [ 973.837328] LustreError: 50616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789367521 with bad export cookie 3788657235635745930 [ 973.838559] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 973.843139] LustreError: 50616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 974.001291] Lustre: server umount lustre-MDT0001 complete [ 986.637353] Lustre: server umount lustre-OST0000 complete [ 998.776937] Lustre: server umount lustre-OST0001 complete [ 1003.044271] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 02:32:30 (1789367550) [ 1004.168340] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1004.914520] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 02:32:32 (1789367552) [ 1009.535047] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 1013.068380] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1022.415233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1022.524290] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1022.568238] Lustre: lustre-MDT0000: new disk, initializing [ 1022.602459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1022.605952] Lustre: Skipped 13 previous similar messages [ 1022.614070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1024.450581] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1029.109621] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1032.227625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1032.365627] Lustre: 54800:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1034.410502] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1034.439376] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1035.176288] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1039.871804] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1040.842367] Lustre: DEBUG MARKER: create test files [ 1055.201239] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1059.384056] Lustre: server umount lustre-OST0000 complete [ 1061.200273] Lustre: server umount lustre-MDT0000 complete [ 1065.499026] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1066.810113] Key type lgssc unregistered [ 1066.976060] LNet: 56426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1066.980323] LNetError: 56426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1066.991557] LNet: Removed LNI 192.168.203.123@tcp [ 1067.423268] Key type .llcrypt unregistered [ 1067.425611] Key type ._llcrypt unregistered [ 1080.030058] Key type ._llcrypt registered [ 1080.032498] Key type .llcrypt registered [ 1080.094409] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1080.647106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1080.655856] alg: No test for adler32 (adler32-zlib) [ 1081.566300] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 1081.690502] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1083.287190] Key type lgssc registered [ 1083.823264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1087.294519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1087.304111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1088.515554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1090.197464] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1093.295684] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1096.315022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1096.473087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1098.535460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 1098.578658] Lustre: 59130:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1098.583505] Lustre: 59130:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1098.587336] Lustre: 59130:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1098.591073] Lustre: 59130:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1098.594982] Lustre: 59130:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1098.599343] Lustre: 59130:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1098.939360] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1101.968032] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1105.224311] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1107.686819] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1114.082039] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1114.082222] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1114.083269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1116.132718] Lustre: server umount lustre-OST0000 complete [ 1117.998887] Lustre: server umount lustre-MDT0000 complete [ 1122.652326] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1123.938197] Key type lgssc unregistered [ 1124.086762] LNet: 60737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1124.091830] LNetError: 60737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1124.103463] LNet: Removed LNI 192.168.203.123@tcp [ 1124.497460] Key type .llcrypt unregistered [ 1124.500873] Key type ._llcrypt unregistered [ 1139.494395] Key type ._llcrypt registered [ 1139.496545] Key type .llcrypt registered [ 1139.550437] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1140.015794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1140.033205] alg: No test for adler32 (adler32-zlib) [ 1140.912263] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 1141.018757] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1142.615196] Key type lgssc registered [ 1143.094420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1146.300728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1146.311529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1147.532668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1149.355794] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1152.526554] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1155.446458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1155.623458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1157.670238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1157.695185] Lustre: 63545:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1157.699240] Lustre: 63545:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1157.701800] Lustre: 63545:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1157.704960] Lustre: 63545:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1157.708820] Lustre: 63545:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1157.712120] Lustre: 63545:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1157.950060] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1161.005377] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1164.448134] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1172.960378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1172.961617] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1172.973569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1174.427675] Lustre: server umount lustre-OST0000 complete [ 1176.126627] Lustre: server umount lustre-MDT0000 complete [ 1180.975897] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1182.189816] Key type lgssc unregistered [ 1182.341759] LNet: 65097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1182.345085] LNetError: 65097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1182.352335] LNet: Removed LNI 192.168.203.123@tcp [ 1182.711836] Key type .llcrypt unregistered [ 1182.713131] Key type ._llcrypt unregistered [ 1189.880192] Key type ._llcrypt registered [ 1189.881388] Key type .llcrypt registered [ 1189.928493] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 1197.539340] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1197.949288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1197.983809] alg: No test for adler32 (adler32-zlib) [ 1198.854743] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 1198.961649] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1200.567291] Key type lgssc registered [ 1201.071697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1217.050051] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1217.063893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1218.182141] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1218.194857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1218.230692] Lustre: lustre-MDT0000: new disk, initializing [ 1218.259696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1218.269798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1219.937665] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1224.771342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1224.826466] Lustre: 68220:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1224.855988] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1224.859117] Lustre: Skipped 1 previous similar message [ 1224.901054] Lustre: lustre-MDT0001: new disk, initializing [ 1224.950080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1224.968418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1224.973963] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1226.669059] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1231.254646] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1233.674384] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1236.492241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1236.630994] Lustre: lustre-OST0000: new disk, initializing [ 1236.634337] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1236.639156] Lustre: 70187:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1236.680634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1237.835970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1237.844654] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1237.896279] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1239.114872] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1243.408627] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1245.081835] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1245.156024] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1246.664540] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1246.745480] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1248.225801] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1248.226464] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1248.228788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1248.231480] Lustre: Skipped 1 previous similar message [ 1253.348483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1253.350697] Lustre: Skipped 2 previous similar messages [ 1253.532197] Lustre: server umount lustre-OST0000 complete [ 1255.904307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1255.911295] LustreError: Skipped 1 previous similar message [ 1255.913989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1255.922104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1258.464088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1258.464711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1263.583923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1263.586236] Lustre: Skipped 2 previous similar messages [ 1269.215138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1269.277386] Lustre: server umount lustre-MDT0000 complete [ 1270.599108] LustreError: 69014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789367818 with bad export cookie 11827359457094570608 [ 1270.602916] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1270.604547] LustreError: 69014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1270.721594] Lustre: server umount lustre-MDT0001 complete [ 1273.882508] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 02:37:01 (1789367821) [ 1278.109755] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1291.773787] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 1295.098821] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1307.051983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1307.137977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1307.149959] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1307.188330] Lustre: lustre-MDT0000: new disk, initializing [ 1307.214208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1307.222318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1308.646660] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1312.883401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1312.927179] Lustre: 76193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1312.955601] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1312.958574] Lustre: Skipped 1 previous similar message [ 1312.998614] Lustre: lustre-MDT0001: new disk, initializing [ 1313.032233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1313.045565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1313.049875] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1314.538304] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1318.359498] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1320.258155] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1322.662850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1322.765254] Lustre: lustre-OST0000: new disk, initializing [ 1322.767463] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1322.771056] Lustre: 78163:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1322.802259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1323.382154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1323.387040] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1323.428078] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1324.826326] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1328.653236] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1330.061537] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1330.126445] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1331.501484] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1331.587719] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1333.730381] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1333.731282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1333.733767] LustreError: Skipped 1 previous similar message [ 1333.738183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1333.740704] Lustre: Skipped 3 previous similar messages [ 1338.378945] Lustre: server umount lustre-OST0000 complete [ 1343.969474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1343.970106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1343.977931] Lustre: Skipped 2 previous similar messages [ 1345.751796] Lustre: server umount lustre-MDT0000 complete [ 1347.054195] LustreError: 76187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789367895 with bad export cookie 11827359457094571490 [ 1347.057347] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1347.058423] LustreError: 76187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1347.165594] Lustre: server umount lustre-MDT0001 complete [ 1350.106260] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 02:38:17 (1789367897) [ 1352.373913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1352.545893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1353.907440] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1356.758173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1358.188765] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1360.683377] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1362.629530] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1364.940332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1366.835620] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1369.297204] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1370.598378] Lustre: 83242:0:(genops.c:1773:obd_export_evict_by_uuid()) MGS: evicting 08b45479-3d45-4d50-8fd8-c022479af3bf at adminstrative request [ 1371.105753] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1371.110643] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 1381.344317] Lustre: MGS: Client 08b45479-3d45-4d50-8fd8-c022479af3bf (at 0@lo) reconnecting [ 1381.347483] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0xa42338c31efa9946 to 0xa42338c31efa9bb5 [ 1381.352102] Lustre: MGC192.168.203.123@tcp: Connection restored to 0@lo (at 0@lo) [ 1403.358808] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1403.443641] Lustre: lustre-OST0001: new disk, initializing [ 1403.446164] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1403.451689] Lustre: 83641:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1403.494153] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1403.497625] Lustre: Skipped 2 previous similar messages [ 1405.539280] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1409.224282] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1410.201945] Lustre: server umount lustre-OST0001 complete [ 1411.565120] LustreError: 82394:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1411.565934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1411.566669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1411.566672] Lustre: Skipped 2 previous similar messages [ 1411.570490] LustreError: 82394:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1411.574752] Lustre: Skipped 1 previous similar message [ 1416.675023] LustreError: 82392:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1416.680070] LustreError: 82392:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1417.486060] Lustre: server umount lustre-OST0000 complete [ 1433.055133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1433.113796] Lustre: server umount lustre-MDT0000 complete [ 1434.275256] LustreError: 80555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789367982 with bad export cookie 11827359457094572981 [ 1434.278653] LustreError: 80555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1434.278824] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1434.390426] Lustre: server umount lustre-MDT0001 complete [ 1437.002905] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 02:39:44 (1789367984) [ 1437.493695] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 1438.028308] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 02:39:45 (1789367985) [ 1438.502292] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 1439.045838] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 02:39:46 (1789367986) [ 1439.530285] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 1440.060704] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 02:39:47 (1789367987) [ 1440.539417] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 1441.055024] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 02:39:48 (1789367988) [ 1441.580888] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 1443.498897] loop: module loaded [ 1443.503350] loop0: detected capacity change from 0 to 4294967296 [ 1443.515081] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 1444.018305] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 1445.062431] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 1446.401820] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 1454.857956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1454.872530] LustreError: 87753:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 1454.875496] LustreError: 87753:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 1454.878556] LustreError: 87753:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 1454.881836] LustreError: 87753:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 1460.812796] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 1463.895261] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1475.482735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1475.557093] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1475.567246] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1475.603870] Lustre: lustre-MDT0000: new disk, initializing [ 1475.623387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1475.629254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1477.017454] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1481.209721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1481.251984] Lustre: 90612:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1481.345401] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1482.645060] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1486.248705] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1487.978670] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1490.226516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1490.346524] Lustre: 92579:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1492.238234] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1495.876750] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1497.183233] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1498.106207] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1498.143551] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1498.266437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1499.450946] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1499.512505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1503.200792] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1503.202088] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1503.204641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1503.208761] Lustre: Skipped 3 previous similar messages [ 1503.212043] Lustre: Skipped 10 previous similar messages [ 1506.185956] Lustre: server umount lustre-OST0000 complete [ 1513.557967] Lustre: server umount lustre-MDT0000 complete [ 1514.711956] LustreError: 90603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368062 with bad export cookie 11827359457094573569 [ 1514.714746] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1514.715732] LustreError: 90603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1514.815293] Lustre: server umount lustre-MDT0001 complete [ 1517.342456] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 02:41:05 (1789368065) [ 1519.553884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1519.703972] LustreError: 94994:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1521.053697] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1523.836153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1525.261157] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1527.546946] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1529.317684] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1531.450220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1533.431022] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1535.811858] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1537.136145] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1545.356422] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 1546.564090] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1546.624759] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1548.627678] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1548.681175] Lustre: lustre-OST0001: new disk, initializing [ 1548.682297] Lustre: Skipped 2 previous similar messages [ 1548.683548] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1548.684887] Lustre: Skipped 3 previous similar messages [ 1548.686459] Lustre: 98421:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1550.519275] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1554.097497] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1555.454444] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1555.963298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1555.966768] Lustre: Skipped 2 previous similar messages [ 1555.968464] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1555.993744] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1556.546735] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1557.890043] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1557.967903] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1568.006701] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1572.859934] Lustre: Failing over lustre-MDT0000 [ 1573.231975] Lustre: server umount lustre-MDT0000 complete [ 1574.790472] LustreError: 94989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1574.794796] LustreError: 94989:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1575.391749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1575.393619] LustreError: Skipped 1 previous similar message [ 1575.394735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1575.397802] Lustre: Skipped 2 previous similar messages [ 1579.903258] LustreError: 97340:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1579.909819] LustreError: 97340:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 1587.823777] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 1587.825323] LDISKFS-fs (dm-0): recovery complete [ 1587.827583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1587.861664] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1587.979093] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1589.260866] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1590.143265] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 1590.144574] Lustre: 94991:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1593.312737] Lustre: 94994:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1593.316722] Lustre: 94994:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 1 previous similar message [ 1593.322900] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1593.339273] LustreError: 101249:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1593.655108] LustreError: 101249:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1593.969305] LustreError: 101249:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1593.973951] LustreError: 101249:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 1 previous similar message [ 1594.287106] LustreError: 101249:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1594.289726] LustreError: 101249:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 1 previous similar message [ 1594.609218] Lustre: 101249:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1594.613161] Lustre: 101249:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 8 previous similar messages [ 1595.241387] LustreError: 101249:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1595.247539] LustreError: 101249:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 3 previous similar messages [ 1595.567100] LustreError: 101249:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1595.569355] LustreError: 101249:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 3 previous similar messages [ 1596.823782] Lustre: 101249:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1596.828135] Lustre: 101249:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 6 previous similar messages [ 1597.455853] LustreError: 101249:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1597.460042] LustreError: 101249:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 6 previous similar messages [ 1597.775084] LustreError: 101249:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1597.777573] LustreError: 101249:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 6 previous similar messages [ 1600.905051] Lustre: 101249:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1600.909449] Lustre: 101249:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 12 previous similar messages [ 1601.537302] LustreError: 101249:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1601.542356] LustreError: 101249:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 12 previous similar messages [ 1601.855105] LustreError: 101249:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1601.857936] LustreError: 101249:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 12 previous similar messages [ 1609.041052] Lustre: 101249:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1609.044036] Lustre: 101249:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 25 previous similar messages [ 1609.673086] LustreError: 101249:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1609.677809] LustreError: 101249:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 25 previous similar messages [ 1609.991119] LustreError: 101249:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1609.993965] LustreError: 101249:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 25 previous similar messages [ 1625.296703] Lustre: 101249:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1625.299929] Lustre: 101249:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 51 previous similar messages [ 1625.929566] LustreError: 101249:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1625.933245] LustreError: 101249:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 51 previous similar messages [ 1626.247098] LustreError: 101249:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1626.249151] LustreError: 101249:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 51 previous similar messages [ 1650.279133] Lustre: 101249:0:(ldlm_lib.c:2428:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 1650.283280] Lustre: 101249:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1650.286725] Lustre: 101249:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client deb2ade5-c094-493c-b2ff-2a0980720ecc@192.168.203.23@tcp [ 1650.292778] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1650.296302] Lustre: 101249:0:(ldlm_lib.c:1948:abort_req_replay_queue()) @@@ aborted: req@00000000aac8958e x1876288066357888/t0(8589934779) o101->deb2ade5-c094-493c-b2ff-2a0980720ecc@192.168.203.23@tcp:89/0 lens 592/0 e 0 to 0 dl 1789368209 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 1650.307961] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1650.311976] Lustre: lustre-MDT0000: Denying connection for new client deb2ade5-c094-493c-b2ff-2a0980720ecc (at 192.168.203.23@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 1650.316395] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 1650.334887] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1650.335164] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 1650.337866] Lustre: Skipped 2 previous similar messages [ 1650.355670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:80 to 0x280000401:577) [ 1650.355695] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:47 to 0x2c0000401:545) [ 1657.823694] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1657.825851] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1657.829288] Lustre: Skipped 3 previous similar messages [ 1657.830979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1657.832992] Lustre: Skipped 4 previous similar messages [ 1663.549731] Lustre: server umount lustre-OST0000 complete [ 1664.993822] LustreError: 100321:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1665.001194] LustreError: 100321:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 1678.815105] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1678.865198] Lustre: server umount lustre-OST0001 complete [ 1686.233270] Lustre: server umount lustre-MDT0000 complete [ 1687.434515] LustreError: 95020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368235 with bad export cookie 11827359457094624753 [ 1687.437546] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1687.438138] LustreError: 95020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1687.559513] Lustre: server umount lustre-MDT0001 complete [ 1690.140200] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 02:43:57 (1789368237) [ 1692.517059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1692.519661] Lustre: *** cfs_fail_loc=197, val=0*** [ 1707.935562] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1713.055324] LustreError: 103221:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.123@tcp: failed processing log, type 4: rc = -110 [ 1718.181884] Lustre: *** cfs_fail_loc=197, val=0*** [ 1718.182977] Lustre: Skipped 65 previous similar messages [ 1738.655313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1738.657345] Lustre: Skipped 7 previous similar messages [ 1740.571428] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1742.815470] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1743.496126] Lustre: server umount lustre-OST0000 complete [ 1746.068382] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 02:44:53 (1789368293) [ 1759.516191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1759.664582] LustreError: 104838:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1759.672311] LustreError: 104838:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 1760.909042] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1764.437489] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1766.361254] LustreError: 104873:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1766.365085] LustreError: 104873:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1766.369931] LustreError: 104873:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 1766.495301] Lustre: server umount lustre-MDT0000 complete [ 1776.524927] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 1779.578596] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1790.692801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1790.794080] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1790.805449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1790.849533] Lustre: lustre-MDT0000: new disk, initializing [ 1790.877840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1792.179406] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1796.069365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1796.107559] Lustre: 109034:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1796.193065] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1797.428553] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1800.991133] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1802.734864] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1804.852490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1804.955576] Lustre: 111002:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1806.816776] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1806.938343] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1810.352051] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1811.780888] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1811.848104] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1813.165683] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1813.238742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1817.056945] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1817.058126] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1817.058719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1817.058724] Lustre: Skipped 12 previous similar messages [ 1817.059313] LustreError: Skipped 1 previous similar message [ 1817.064986] Lustre: Skipped 6 previous similar messages [ 1828.319072] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1828.365583] Lustre: server umount lustre-OST0000 complete [ 1835.667527] Lustre: server umount lustre-MDT0000 complete [ 1836.820979] LustreError: 109029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368384 with bad export cookie 11827359457094635421 [ 1836.823972] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1836.824537] LustreError: 109029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1836.935150] Lustre: server umount lustre-MDT0001 complete [ 1839.538425] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 02:46:27 (1789368387) [ 1842.615721] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1843.645495] Key type lgssc unregistered [ 1843.771273] LNet: 113486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1843.773508] LNetError: 113486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1843.783286] LNet: Removed LNI 192.168.203.123@tcp [ 1844.059032] Key type .llcrypt unregistered [ 1844.060043] Key type ._llcrypt unregistered [ 1849.397142] Key type ._llcrypt registered [ 1849.398870] Key type .llcrypt registered [ 1849.441578] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 1855.588543] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1855.895984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1855.904751] alg: No test for adler32 (adler32-zlib) [ 1856.759255] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 1856.850930] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1858.431258] Key type lgssc registered [ 1858.815773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1871.225420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1871.229923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1872.306372] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1872.316474] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1872.349712] Lustre: lustre-MDT0000: new disk, initializing [ 1872.365791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1872.371067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1873.641860] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1877.638511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1877.740909] Lustre: lustre-OST0000: new disk, initializing [ 1877.742646] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1877.744311] Lustre: Skipped 1 previous similar message [ 1877.746217] Lustre: 117627:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1877.768741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1879.683376] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1880.701756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1880.704777] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1880.714190] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1883.746203] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1883.798945] Lustre: lustre-OST0001: new disk, initializing [ 1883.800842] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1883.803921] Lustre: 118489:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1883.827280] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1885.678774] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1894.368487] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1894.369298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1894.371693] Lustre: Skipped 1 previous similar message [ 1894.374670] Lustre: Skipped 1 previous similar message [ 1895.721135] Lustre: server umount lustre-MDT0000 complete [ 1897.686692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1899.024975] LustreError: 116811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368447 with bad export cookie 17366253436162667794 [ 1899.026737] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1899.027622] LustreError: 116811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1899.055153] Lustre: server umount lustre-OST0000 complete [ 1900.306398] Lustre: server umount lustre-OST0001 complete [ 1902.993805] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 02:47:30 (1789368450) [ 1909.932885] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 02:47:37 (1789368457) [ 1912.708151] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1919.311321] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 02:47:47 (1789368467) [ 1931.489320] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 1934.975312] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1946.955426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1947.037848] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1947.048192] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1947.086821] Lustre: lustre-MDT0000: new disk, initializing [ 1947.108940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1947.115393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1948.581512] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1952.682069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1952.727317] Lustre: 126221:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1952.752797] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1952.754663] Lustre: Skipped 1 previous similar message [ 1952.792881] Lustre: lustre-MDT0001: new disk, initializing [ 1952.826235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1952.837650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1952.844164] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1954.193963] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1957.980220] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1959.747519] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1962.023176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1962.133554] Lustre: lustre-OST0000: new disk, initializing [ 1962.135347] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1962.138407] Lustre: 128189:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1962.167515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1964.040763] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1967.501782] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1968.627796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1968.632721] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1968.668301] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1968.803349] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1968.867307] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1970.091066] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1970.155741] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1973.731282] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1973.731992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1973.738686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1973.741203] Lustre: Skipped 1 previous similar message [ 1976.850451] Lustre: server umount lustre-OST0000 complete [ 1978.335895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1978.338427] LustreError: Skipped 1 previous similar message [ 1978.339892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1978.344264] Lustre: Skipped 1 previous similar message [ 1978.346109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1983.968131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1983.970205] Lustre: Skipped 2 previous similar messages [ 1984.214617] Lustre: server umount lustre-MDT0000 complete [ 1985.437207] LustreError: 126214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368533 with bad export cookie 17366253436162668928 [ 1985.440732] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1985.441453] LustreError: 126214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1985.553154] Lustre: server umount lustre-MDT0001 complete [ 1988.282562] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 02:48:56 (1789368536) [ 1990.485867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1990.675337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1992.037587] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1994.952745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1996.460931] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1998.826313] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2000.597783] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2002.922287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2004.924984] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2007.321013] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2019.782154] Lustre: *** cfs_fail_loc=159, val=0*** [ 2036.113474] Lustre: lustre-MDT0000: Client a70fc81f-3c94-41f5-9d6f-93a740feecff (at 192.168.203.23@tcp) reconnecting [ 2036.615063] Lustre: *** cfs_fail_loc=159, val=0*** [ 2036.616574] Lustre: Skipped 5 previous similar messages [ 2051.982376] Lustre: lustre-MDT0000: Client a70fc81f-3c94-41f5-9d6f-93a740feecff (at 192.168.203.23@tcp) reconnecting [ 2056.673771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2056.674301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2056.676118] LustreError: Skipped 1 previous similar message [ 2056.676899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2056.679903] Lustre: Skipped 2 previous similar messages [ 2059.281889] Lustre: server umount lustre-OST0000 complete [ 2061.793063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2061.793163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2061.798484] Lustre: Skipped 1 previous similar message [ 2066.708320] Lustre: server umount lustre-MDT0000 complete [ 2066.913243] LustreError: 131329:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2066.919422] LustreError: 131329:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2067.992593] LustreError: 130581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368616 with bad export cookie 17366253436162669796 [ 2067.997164] LustreError: 130581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2067.997834] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2068.114467] Lustre: server umount lustre-MDT0001 complete [ 2072.029996] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2073.107794] Key type lgssc unregistered [ 2073.230357] LNet: 134392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2073.233325] LNetError: 134392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2073.240241] LNet: Removed LNI 192.168.203.123@tcp [ 2073.540391] Key type .llcrypt unregistered [ 2073.541445] Key type ._llcrypt unregistered [ 2082.624814] Key type ._llcrypt registered [ 2082.625783] Key type .llcrypt registered [ 2082.663692] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2083.083185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2083.090827] alg: No test for adler32 (adler32-zlib) [ 2083.936286] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 2084.022803] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2085.607185] Key type lgssc registered [ 2085.975387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2088.586211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2088.591883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2089.726434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2091.153451] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2094.028303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2094.170253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2095.615675] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2098.137353] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2099.884207] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2102.142987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2102.258924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2104.296044] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2106.827683] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2115.559105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 2122.072625] Lustre: *** cfs_fail_loc=159, val=0*** [ 2138.509286] Lustre: lustre-MDT0000: Client e30f3009-6dd5-408d-8add-ff7c9068f936 (at 192.168.203.23@tcp) reconnecting [ 2139.249892] Lustre: *** cfs_fail_loc=159, val=0*** [ 2139.253207] Lustre: Skipped 1 previous similar message [ 2155.402821] Lustre: lustre-MDT0001: Client e30f3009-6dd5-408d-8add-ff7c9068f936 (at 192.168.203.23@tcp) reconnecting [ 2155.947501] Lustre: *** cfs_fail_loc=159, val=0*** [ 2155.948871] Lustre: Skipped 2 previous similar messages [ 2172.303158] Lustre: lustre-MDT0001: Client e30f3009-6dd5-408d-8add-ff7c9068f936 (at 192.168.203.23@tcp) reconnecting [ 2173.469466] Lustre: 138834:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2174.261632] Lustre: *** cfs_fail_loc=159, val=0*** [ 2174.264360] Lustre: Skipped 3 previous similar messages [ 2190.732360] Lustre: lustre-MDT0000: Client 27e7e6bc-3826-4e46-9409-798a146b902f (at 192.168.203.23@tcp) reconnecting [ 2191.301880] Lustre: *** cfs_fail_loc=159, val=0*** [ 2191.304537] Lustre: Skipped 15 previous similar messages [ 2207.629666] Lustre: lustre-MDT0000: Client 27e7e6bc-3826-4e46-9409-798a146b902f (at 192.168.203.23@tcp) reconnecting [ 2208.156027] Lustre: 139126:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2211.808751] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2211.812113] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2211.817967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2212.835775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2212.836152] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2212.836556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2212.837573] Lustre: Skipped 1 previous similar message [ 2212.845959] LustreError: Skipped 1 previous similar message [ 2215.502400] Lustre: server umount lustre-OST0000 complete [ 2216.927698] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2216.930036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2216.933650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2223.008542] Lustre: server umount lustre-MDT0000 complete [ 2223.072927] LustreError: 138895:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2223.080525] LustreError: 138895:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2224.229226] LustreError: 135760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368772 with bad export cookie 1177610703375200761 [ 2224.232676] LustreError: 135760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2224.233618] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2224.344412] Lustre: server umount lustre-MDT0001 complete [ 2228.104893] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2229.147433] Key type lgssc unregistered [ 2229.279457] LNet: 140175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2229.283754] LNetError: 140175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2229.292447] LNet: Removed LNI 192.168.203.123@tcp [ 2229.652094] Key type .llcrypt unregistered [ 2229.653815] Key type ._llcrypt unregistered [ 2238.655106] Key type ._llcrypt registered [ 2238.656087] Key type .llcrypt registered [ 2238.690782] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2239.007892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2239.058313] alg: No test for adler32 (adler32-zlib) [ 2239.904930] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 2239.989649] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2241.567192] Key type lgssc registered [ 2241.932924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2244.580629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2244.586211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2245.724978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2247.120937] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2250.069104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2250.198119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2251.565887] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2254.016981] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2255.901738] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2258.159232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2258.285888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2260.182274] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2260.327851] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 2262.614552] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2270.568439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 2275.271209] Lustre: 144165:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2275.929680] Lustre: 144213:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2276.413262] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 2277.557078] Lustre: 144359:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 2277.863100] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 2277.867519] Lustre: Skipped 1 previous similar message [ 2278.170788] Lustre: 144457:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2280.930773] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2280.932155] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2280.934670] LustreError: Skipped 1 previous similar message [ 2280.940165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2280.943507] Lustre: Skipped 1 previous similar message [ 2285.258688] Lustre: server umount lustre-OST0000 complete [ 2291.169331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2291.169452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2291.170590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2291.173220] Lustre: Skipped 2 previous similar messages [ 2292.698644] Lustre: server umount lustre-MDT0000 complete [ 2293.951174] LustreError: 141586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368842 with bad export cookie 5527983995247432085 [ 2293.955623] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2293.957770] LustreError: 141586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2294.075793] Lustre: server umount lustre-MDT0001 complete [ 2297.730316] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2298.759739] Key type lgssc unregistered [ 2298.897448] LNet: 145512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2298.900889] LNetError: 145512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2298.910357] LNet: Removed LNI 192.168.203.123@tcp [ 2299.221822] Key type .llcrypt unregistered [ 2299.223032] Key type ._llcrypt unregistered [ 2308.511392] Key type ._llcrypt registered [ 2308.512830] Key type .llcrypt registered [ 2308.549338] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2308.844923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2308.855755] alg: No test for adler32 (adler32-zlib) [ 2309.705447] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 2309.788736] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2311.367203] Key type lgssc registered [ 2311.724153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2314.305675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2314.312284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2315.462403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2316.802510] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2319.646554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2319.782131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2321.216900] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2323.631671] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2325.519418] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2327.746491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2327.883405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2328.934084] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 2329.808165] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2332.160897] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2341.222529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 2344.905845] Lustre: *** cfs_fail_loc=159, val=0*** [ 2361.228552] Lustre: lustre-MDT0000: Client 752e4af1-c87a-46d4-bf78-8100f365477f (at 192.168.203.23@tcp) reconnecting [ 2364.895917] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2364.898195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2364.902286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2366.944723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2366.948408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2366.953859] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2368.590286] Lustre: server umount lustre-OST0000 complete [ 2371.040036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2371.042252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2371.046083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2371.047715] Lustre: Skipped 1 previous similar message [ 2376.026578] Lustre: server umount lustre-MDT0000 complete [ 2377.184140] LustreError: 146898:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2377.188773] LustreError: 146898:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2377.268256] LustreError: 146879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368925 with bad export cookie 3802960981638543125 [ 2377.272135] LustreError: 146879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2377.273337] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2377.395907] Lustre: server umount lustre-MDT0001 complete [ 2381.054657] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2382.010316] Key type lgssc unregistered [ 2382.136341] LNet: 150580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2382.138713] LNetError: 150580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2382.147350] LNet: Removed LNI 192.168.203.123@tcp [ 2382.450963] Key type .llcrypt unregistered [ 2382.451990] Key type ._llcrypt unregistered [ 2391.463516] Key type ._llcrypt registered [ 2391.464570] Key type .llcrypt registered [ 2391.501995] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2391.910975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2391.932620] alg: No test for adler32 (adler32-zlib) [ 2392.775932] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 2392.864318] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2394.439203] Key type lgssc registered [ 2394.786744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2397.279380] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2397.286540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2398.424476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2399.714341] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2402.556547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2402.692268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2403.943208] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2406.313629] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2408.176686] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2410.440647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2410.590488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2412.549229] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2412.652097] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 2415.044820] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2424.948595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 2438.559787] Lustre: lustre-OST0000: evicting 9022f842-7d1d-441e-b55c-b4d7d28bae00 (at 192.168.203.23@tcp) by administrative request [ 2438.563409] Lustre: Skipped 1 previous similar message [ 2464.224451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2464.227979] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2464.235148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2466.274877] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2466.275071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2466.275223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2468.613537] Lustre: server umount lustre-OST0000 complete [ 2471.393087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2471.393698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2471.399891] Lustre: Skipped 1 previous similar message [ 2471.402856] Lustre: Skipped 1 previous similar message [ 2476.512074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2476.514630] Lustre: Skipped 1 previous similar message [ 2481.631983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2481.633777] Lustre: Skipped 1 previous similar message [ 2484.191095] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2484.247913] Lustre: server umount lustre-MDT0000 complete [ 2485.464026] LustreError: 151946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369033 with bad export cookie 5787976613248160827 [ 2485.467149] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2485.468839] LustreError: 151946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2485.583240] Lustre: server umount lustre-MDT0001 complete [ 2489.120684] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2490.101790] Key type lgssc unregistered [ 2490.245358] LNet: 156054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2490.248254] LNetError: 156054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2490.259334] LNet: Removed LNI 192.168.203.123@tcp [ 2490.552323] Key type .llcrypt unregistered [ 2490.553357] Key type ._llcrypt unregistered [ 2515.824924] Key type ._llcrypt registered [ 2515.826013] Key type .llcrypt registered [ 2515.872349] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2516.287654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2516.356794] alg: No test for adler32 (adler32-zlib) [ 2517.249799] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 2517.357896] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2518.951282] Key type lgssc registered [ 2519.363932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2522.198314] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2522.204884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2523.341271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2524.634258] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2527.596810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2527.731786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2529.054069] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2531.977088] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2533.968671] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2536.182592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2536.309081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2537.318365] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 2538.070403] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2540.598108] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2548.587169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2557.919879] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2557.923070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2557.928367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2558.944987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2558.945043] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2558.946561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2558.947443] Lustre: Skipped 1 previous similar message [ 2564.064356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2564.067299] Lustre: Skipped 1 previous similar message [ 2568.159103] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2568.216267] Lustre: server umount lustre-OST0000 complete [ 2573.791836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2573.796757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2573.801844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2575.575747] Lustre: server umount lustre-MDT0000 complete [ 2576.806244] LustreError: 160359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369124 with bad export cookie 6279732623324075654 [ 2576.810123] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2576.934021] Lustre: server umount lustre-MDT0001 complete [ 2580.621411] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2581.605146] Key type lgssc unregistered [ 2581.748390] LNet: 162071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2581.750703] LNetError: 162071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2581.760261] LNet: Removed LNI 192.168.203.123@tcp [ 2582.077046] Key type .llcrypt unregistered [ 2582.078309] Key type ._llcrypt unregistered [ 2591.570699] Key type ._llcrypt registered [ 2591.571686] Key type .llcrypt registered [ 2591.607678] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2591.947481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2591.981405] alg: No test for adler32 (adler32-zlib) [ 2592.841525] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 2592.930877] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2594.511157] Key type lgssc registered [ 2594.867156] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2599.779810] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 02:59:07 (1789369147) [ 2604.918497] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 2608.081333] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2620.419936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2620.426385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2621.505882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2621.516620] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2621.549117] Lustre: lustre-MDT0000: new disk, initializing [ 2621.569413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2621.576219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2622.841982] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2626.827082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2626.866072] Lustre: 166299:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2626.886749] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2626.888509] Lustre: Skipped 1 previous similar message [ 2626.921832] Lustre: lustre-MDT0001: new disk, initializing [ 2626.950323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2626.961184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2626.964579] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2628.185220] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2631.694629] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2633.417835] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2635.561905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2635.651427] Lustre: lustre-OST0000: new disk, initializing [ 2635.652880] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2635.655638] Lustre: 168266:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2635.682397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2636.149372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2636.153829] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2636.197404] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2637.470260] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2640.873480] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2646.498344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2646.499221] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2646.504760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2646.506939] LustreError: Skipped 1 previous similar message [ 2646.512264] Lustre: Skipped 1 previous similar message [ 2649.936613] Lustre: server umount lustre-OST0000 complete [ 2651.616364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2651.617194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2651.620027] Lustre: Skipped 2 previous similar messages [ 2656.736175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2656.738552] Lustre: Skipped 2 previous similar messages [ 2657.364293] Lustre: server umount lustre-MDT0000 complete [ 2658.549253] LustreError: 166291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369206 with bad export cookie 10176669064378416047 [ 2658.552251] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2658.555249] LustreError: 166291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2658.671509] Lustre: server umount lustre-MDT0001 complete [ 2662.408114] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2663.419267] Key type lgssc unregistered [ 2663.547335] LNet: 170167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2663.549566] LNetError: 170167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2663.556907] LNet: Removed LNI 192.168.203.123@tcp [ 2663.878264] Key type .llcrypt unregistered [ 2663.879427] Key type ._llcrypt unregistered [ 2672.961954] Key type ._llcrypt registered [ 2672.963050] Key type .llcrypt registered [ 2673.007055] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2673.365464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2673.418063] alg: No test for adler32 (adler32-zlib) [ 2674.274806] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 2674.363457] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2675.943173] Key type lgssc registered [ 2676.287727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2678.911557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2678.919735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2680.064385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2681.376996] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2684.210601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2684.345825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2685.797952] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2688.166307] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2689.810906] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2691.878270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2691.990867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2693.859850] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2696.296941] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2705.318652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2709.168589] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2709.178028] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2709.184447] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2709.188282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2709.192551] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2709.539093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2709.898301] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2709.901440] Lustre: Skipped 1 previous similar message [ 2709.909184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2709.913158] Lustre: Skipped 1 previous similar message [ 2709.914976] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2709.917442] Lustre: Skipped 1 previous similar message [ 2709.919387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2709.924978] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2709.927931] Lustre: Skipped 1 previous similar message [ 2710.236476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2710.555330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2710.863525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2711.190602] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2711.193123] Lustre: Skipped 3 previous similar messages [ 2711.199679] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2711.203974] Lustre: Skipped 3 previous similar messages [ 2711.205673] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2711.207645] Lustre: Skipped 3 previous similar messages [ 2711.210044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2711.213288] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2711.215087] Lustre: Skipped 3 previous similar messages [ 2711.521310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2711.822767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2712.131354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2712.484334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2712.809434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2713.121306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2713.419334] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2713.422393] Lustre: Skipped 6 previous similar messages [ 2713.428149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2713.432403] Lustre: Skipped 6 previous similar messages [ 2713.433948] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2713.436752] Lustre: Skipped 6 previous similar messages [ 2713.439532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2713.443139] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2713.445024] Lustre: Skipped 6 previous similar messages [ 2713.761095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2714.064421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2714.353437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2714.647723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2714.969079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2715.286576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2715.599458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2715.910497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2716.222291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2716.527152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2716.822669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2717.129169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2717.447670] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2717.449991] Lustre: Skipped 12 previous similar messages [ 2717.456379] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2717.461091] Lustre: Skipped 12 previous similar messages [ 2717.462743] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2717.465344] Lustre: Skipped 12 previous similar messages [ 2717.467086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2717.470530] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2717.472749] Lustre: Skipped 12 previous similar messages [ 2717.779672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2718.085547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2718.402752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2718.729911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2719.029927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2719.326486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2719.620894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2719.935050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2720.227499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2720.539516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2720.847739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2721.167323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2721.465443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2721.771640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2722.071180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2722.386623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2722.689597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2722.986804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2723.283988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2723.596761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2723.898349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2724.201200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2724.510345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2724.827916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2725.124155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2725.425219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2725.727538] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2725.730400] Lustre: Skipped 26 previous similar messages [ 2725.736903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2725.740368] Lustre: Skipped 26 previous similar messages [ 2725.742143] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2725.744967] Lustre: Skipped 26 previous similar messages [ 2725.746427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2725.751095] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2725.753425] Lustre: Skipped 26 previous similar messages [ 2726.053598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2726.348536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2726.643958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2726.950807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2727.259253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2727.549392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2727.850082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2728.151378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2728.478238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2728.796492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2729.094487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2729.390676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2729.708365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2729.995963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2730.290519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2730.582958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2730.904139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2731.211758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2731.509068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2731.803558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2732.118219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2732.410729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2732.704227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2732.997966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2733.297595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2733.599532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2733.916994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2734.222534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2734.534630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2734.819593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2735.126721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2735.432790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2735.772587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2736.067829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2736.361088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2736.655774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2736.966450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2737.259578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2737.546688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2737.829839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2738.125401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2738.413231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2738.709230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2739.011585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2739.308432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2739.599449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2739.904764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2740.204057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2740.507932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2740.790233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2741.089462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2741.382685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2741.672986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2741.951410] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2741.953603] Lustre: Skipped 53 previous similar messages [ 2741.958467] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2741.961937] Lustre: Skipped 53 previous similar messages [ 2741.963589] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2741.966853] Lustre: Skipped 53 previous similar messages [ 2741.969084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2741.973510] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2741.975239] Lustre: Skipped 53 previous similar messages [ 2742.260656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2742.549196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2742.848495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2743.122686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2743.408509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2743.704562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2744.002633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2744.292641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2744.584086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2744.871795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2745.169176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2745.465167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2745.760423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2746.058066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2746.367732] LustreError: 171573:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2746.370692] LustreError: 171573:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2746.371598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2746.663483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2746.957220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2747.252217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2747.558984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2747.852976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2748.150282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2748.446853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2748.756328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2749.049690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2749.351081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2749.643247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2749.941454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2750.230122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2750.523297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2750.814072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2751.128254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2751.438514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2751.739584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2752.036200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2752.347084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2752.646891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2752.963404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2753.258595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2753.564151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2753.853439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2754.154066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2754.448804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2754.756745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2755.051245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2755.346917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2755.637257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2755.972082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2756.276325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2756.578423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2756.882212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2757.215520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2757.518900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2757.812222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2758.125258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2758.432441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2758.732673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2759.055292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2759.360406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2759.676952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2759.983818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2760.279931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2760.561730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2760.890921] LustreError: 171573:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2760.895130] LustreError: 171573:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2760.895314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2761.211454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2761.524124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2761.824873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2762.142068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2762.435710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2762.731313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2763.034602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2763.339229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2763.622330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2763.930708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2764.223340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2764.537414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2764.839717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2765.127585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2765.418938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2765.725388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2766.023154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2766.328360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2766.638496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2766.985557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2767.302206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2767.614888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2767.911874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2768.223538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2768.507904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2768.806794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2769.102888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2769.421105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2769.721890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2770.038164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2770.336070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2770.640994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2770.940331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2771.228168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2771.516049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2771.822776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2772.101818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2772.398754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2772.700243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2773.013333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2773.311522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2773.612346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2773.920781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2774.245901] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2774.248359] Lustre: Skipped 106 previous similar messages [ 2774.254850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2774.259922] Lustre: Skipped 106 previous similar messages [ 2774.261971] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2774.264191] Lustre: Skipped 106 previous similar messages [ 2774.265701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2774.269077] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2774.270729] Lustre: Skipped 106 previous similar messages [ 2774.570119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2774.882114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2775.186542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2775.504849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2775.815242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2776.138094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2776.451451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2776.781847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2777.090463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2777.405637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2777.708770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2778.034961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2778.336839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2778.633963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2778.934976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2779.249858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2779.545783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2779.835932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2780.137434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2780.448285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2780.741085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2781.034978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2781.333600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2781.645921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2781.943868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2782.250651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2782.557357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2782.872737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2783.164538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2783.468294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2783.770335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2784.064363] LustreError: 171573:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2784.068268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2784.070039] LustreError: 171573:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2784.355171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2784.658684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2784.947407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2785.257662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2785.545442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2785.834261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2786.156281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2786.487728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2786.791944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2787.095351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2787.394256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2787.695868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2787.992127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2788.304189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2788.594617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2788.896227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2789.195281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2789.501733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2789.821818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2790.131392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2790.445135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2790.778434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2791.072795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2791.389335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2791.682251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2791.982649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2792.277260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2792.570847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2792.869541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2793.165986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2793.482205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2793.791589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2794.095158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2794.397217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2794.697173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2794.997492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2795.291871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2795.599217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2795.888876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2796.210492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2796.505055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2796.805259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2797.101417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2797.416898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2797.714406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2798.015452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2798.321035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2798.630770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2798.951859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2799.247372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2799.548499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2799.851447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2800.143228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2800.445656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2800.739436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2801.043281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2801.335516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2801.628274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2801.930733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2802.241744] LustreError: 171573:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2802.244687] LustreError: 171573:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2802.245656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2802.548146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2802.851147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2803.150162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2803.492732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2803.810513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2804.116725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2804.420154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2804.757831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2805.076753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2805.385871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2805.690304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2806.037901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2806.358747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2806.692552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2807.013967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2807.351635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2807.670320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2807.971074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2808.282400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2808.621847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2808.925235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2809.228409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2809.528991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2809.854354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2810.168183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2810.472291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2810.770382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2811.090165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2811.392780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2811.684989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2811.983936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2812.282139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2812.573509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2812.882600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2813.181576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2813.496848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2813.793925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2814.093369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2814.381419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2814.686599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2814.984272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2815.280658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2815.584417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2815.897985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2816.195786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2816.495058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2816.799066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2817.115812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2817.414620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2817.708363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2818.004904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2818.314469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2818.612234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2818.914873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2819.217245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2819.536927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2819.833696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2820.134450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2820.437899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2820.744199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2821.036371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2821.338154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2821.636344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2821.954476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2822.269304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2822.565686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2822.860357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2823.172231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2823.468944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2823.796179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2824.109415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2824.434405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2824.740689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2825.046185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2825.353532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2825.666094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2825.965301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2826.263617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2826.563806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2826.906673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2827.211551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2827.521409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2827.826873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2828.151488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2828.453368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2828.757494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2829.070618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2829.397256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2829.696273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2829.987319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2830.285323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2830.597944] LustreError: 171573:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2830.602768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2830.608104] LustreError: 171573:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2830.911935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2831.247246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2831.532852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2831.821795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2832.100404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2832.394717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2832.698628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2833.008331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2833.305074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2833.612764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2833.974295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2847.712155] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2847.715927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2847.719470] Lustre: Skipped 195 previous similar messages [ 2847.721799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2848.059257] Lustre: server umount lustre-OST0000 complete [ 2853.344095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2853.347050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2858.976811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2858.979768] Lustre: Skipped 2 previous similar messages [ 2859.027101] Lustre: server umount lustre-MDT0000 complete [ 2860.272434] LustreError: 171533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369408 with bad export cookie 7240135446556827253 [ 2860.277422] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2860.413783] Lustre: server umount lustre-MDT0001 complete [ 2864.261829] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2865.387226] Key type lgssc unregistered [ 2865.528715] LNet: 196401:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2865.534788] LNetError: 196401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2865.543507] LNet: Removed LNI 192.168.203.123@tcp [ 2865.889552] Key type .llcrypt unregistered [ 2865.891055] Key type ._llcrypt unregistered [ 2875.144369] Key type ._llcrypt registered [ 2875.145414] Key type .llcrypt registered [ 2875.178456] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2875.481962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2875.516044] alg: No test for adler32 (adler32-zlib) [ 2876.360225] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 2876.444263] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2878.023138] Key type lgssc registered [ 2878.373458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2880.755135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2880.760548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2881.902077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2883.215631] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2886.019198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2886.147155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2887.611498] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2890.060304] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2892.016879] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2894.362697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2894.485681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2896.433856] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2898.970708] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2907.812534] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8723 to 0x280000401:8769) [ 2911.442720] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2912.062218] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2912.068459] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2912.073542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2912.077738] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2912.077772] LustreError: 197808:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 2913.250695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2913.251360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2913.257104] Lustre: Skipped 1 previous similar message [ 2914.692584] Lustre: lustre-OST0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 2914.694881] Lustre: Skipped 1 previous similar message [ 2914.771650] Lustre: server umount lustre-OST0000 complete [ 2932.159060] LustreError: 197808:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 2942.630010] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2942.735484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2944.600480] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2946.933168] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2952.996796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2953.001844] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2953.001996] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2953.008693] Lustre: Skipped 1 previous similar message [ 2956.640742] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 9 sec [ 2958.537867] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff93020607b000.ost_server_uuid 50 [ 2959.076453] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff93020607b000.ost_server_uuid in FULL state after 0 sec [ 2963.424864] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2963.426922] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2963.427784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2963.438524] Lustre: Skipped 1 previous similar message [ 2966.155425] Lustre: server umount lustre-OST0000 complete [ 2968.031952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2968.035194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2968.039947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2968.042566] Lustre: Skipped 1 previous similar message [ 2973.520483] Lustre: server umount lustre-MDT0000 complete [ 2973.664097] LustreError: 197787:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2973.669344] LustreError: 197787:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2974.739226] LustreError: 197768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369522 with bad export cookie 16530824376009970727 [ 2974.742057] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2974.743873] LustreError: 197768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2974.857509] Lustre: server umount lustre-MDT0001 complete [ 2978.560932] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2979.552187] Key type lgssc unregistered [ 2979.690409] LNet: 203013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2979.693570] LNetError: 203013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2979.701264] LNet: Removed LNI 192.168.203.123@tcp [ 2980.016826] Key type .llcrypt unregistered [ 2980.017953] Key type ._llcrypt unregistered [ 2991.095052] Key type ._llcrypt registered [ 2991.096131] Key type .llcrypt registered [ 2991.132039] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2991.436107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2991.443390] alg: No test for adler32 (adler32-zlib) [ 2992.290386] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 2992.380740] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2993.959173] Key type lgssc registered [ 2994.334553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2997.453679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2997.463210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2998.616375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2999.953359] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3002.791859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3002.918983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3004.232126] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3005.179943] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3007.302757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3007.425630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3008.497104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8771 to 0x280000401:8801) [ 3009.876280] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3012.668605] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3012.731406] Lustre: lustre-OST0001: new disk, initializing [ 3012.733808] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3012.736748] Lustre: 207024:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3012.763230] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3014.651218] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3015.296419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3015.300775] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3015.332382] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3020.009223] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3025.650150] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3033.567909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3033.570471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3033.575216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3036.129583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3036.129962] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3036.135620] Lustre: Skipped 2 previous similar messages [ 3037.980255] Lustre: server umount lustre-MDT0000 complete [ 3039.132135] LustreError: 204694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369587 with bad export cookie 12116742120138906768 [ 3039.135282] LustreError: 204694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3039.135413] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3039.244451] Lustre: server umount lustre-MDT0001 complete [ 3050.434417] Lustre: server umount lustre-OST0000 complete [ 3061.853640] Lustre: server umount lustre-OST0001 complete [ 3068.728501] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3072.497741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3072.671723] LustreError: 210129:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3072.680980] LustreError: 210129:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3072.690125] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3074.052735] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3076.947910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3078.415281] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3079.395486] Lustre: 211269:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3081.583990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3081.726736] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3081.728754] Lustre: Skipped 1 previous similar message [ 3082.787278] LustreError: 211623:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3082.792885] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8803 to 0x280000401:8833) [ 3083.768641] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3086.592403] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3087.722250] Lustre: mylustre-OST0001: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3088.616583] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3091.752313] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3092.971862] Lustre: 213139:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3097.568127] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3097.571881] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3097.577660] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3097.580223] Lustre: Skipped 3 previous similar messages [ 3102.430422] Lustre: server umount mylustre-MDT0000 complete [ 3103.205318] LustreError: 210130:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3103.216310] LustreError: 210130:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3103.666450] LustreError: 210105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369651 with bad export cookie 12116742120138909533 [ 3103.669807] LustreError: 210105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3103.670301] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3103.780535] Lustre: server umount mylustre-MDT0001 complete [ 3115.524034] Lustre: server umount mylustre-OST0000 complete [ 3126.039160] Lustre: server umount mylustre-OST0001 complete [ 3132.976174] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3136.329723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3136.473031] LustreError: 215444:0:(ldlm_lib.c:1199:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3136.490488] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3136.492558] Lustre: Skipped 1 previous similar message [ 3137.934121] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3141.096487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3142.552931] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3143.509691] Lustre: 216586:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3145.666385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3146.851201] LustreError: 216940:0:(ldlm_lib.c:1199:target_handle_connect()) tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3146.854749] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8803 to 0x280000401:8865) [ 3147.755867] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3150.597022] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3151.722563] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3152.558685] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3155.726975] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3156.937625] Lustre: 218454:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3161.568059] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3161.571515] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3161.575336] Lustre: Skipped 3 previous similar messages [ 3161.577199] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 3161.578965] Lustre: Skipped 4 previous similar messages [ 3165.032889] Lustre: server umount tfs-MDT0000 complete [ 3166.276257] LustreError: 215419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369714 with bad export cookie 12116742120138911423 [ 3166.279571] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3166.279773] LustreError: 215419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3166.383859] Lustre: server umount tfs-MDT0001 complete [ 3177.858451] Lustre: server umount tfs-OST0000 complete [ 3189.340138] Lustre: server umount tfs-OST0001 complete [ 3195.918431] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3199.696139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3199.862604] LustreError: 220699:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3199.883328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3199.886033] Lustre: Skipped 3 previous similar messages [ 3201.207094] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3204.167676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3205.645791] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3206.604463] Lustre: 221841:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3208.803719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3209.958727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8803 to 0x280000401:8897) [ 3210.821819] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3213.547796] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3214.635554] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3215.443142] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3218.418465] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3219.561246] Lustre: 223709:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3221.643948] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 03:09:29 (1789369769) [ 3235.302371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3235.303264] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3235.306048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3235.306062] Lustre: Skipped 4 previous similar messages [ 3235.313731] Lustre: Skipped 6 previous similar messages [ 3239.600609] Lustre: server umount lustre-MDT0000 complete [ 3240.417662] LustreError: 220700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3240.425383] LustreError: 220700:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3240.903141] LustreError: 220676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369788 with bad export cookie 12116742120138913089 [ 3240.907144] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3240.908356] LustreError: 220676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3245.537880] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3245.539813] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3245.549262] Lustre: Skipped 1 previous similar message [ 3245.555290] Lustre: Skipped 4 previous similar messages [ 3255.263095] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3255.350450] Lustre: server umount lustre-MDT0001 complete [ 3262.650230] Lustre: server umount lustre-OST0000 complete [ 3278.303127] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3278.363326] Lustre: server umount lustre-OST0001 complete [ 3280.272700] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 3283.178151] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3295.071393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3295.170366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3295.182518] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3295.223739] Lustre: lustre-MDT0000: new disk, initializing [ 3295.246450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 3295.248837] Lustre: Skipped 3 previous similar messages [ 3295.254045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3296.701831] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3300.792056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3300.833400] Lustre: 226886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3300.859962] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3300.861874] Lustre: Skipped 1 previous similar message [ 3300.897996] Lustre: lustre-MDT0001: new disk, initializing [ 3300.968476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3300.974570] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3302.445199] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3306.622073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3306.722478] Lustre: lustre-OST0000: new disk, initializing [ 3306.724271] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3306.727840] Lustre: 228468:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3308.534885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3308.537622] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3308.578286] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3308.975360] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3313.220164] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3313.285765] Lustre: lustre-OST0001: new disk, initializing [ 3313.288061] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3313.290631] Lustre: 229337:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3314.634967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3314.638469] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3314.678258] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3315.325225] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3319.884519] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 03:11:07 (1789369867) [ 3323.277992] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 03:11:11 (1789369871) [ 3335.141420] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3335.148461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3335.151486] Lustre: Skipped 5 previous similar messages [ 3340.803366] Lustre: server umount lustre-OST0000 complete [ 3345.377641] LustreError: 228454:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3345.383726] LustreError: 228454:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 3348.779419] Lustre: server umount lustre-MDT0000 complete [ 3350.195156] LustreError: 226881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369898 with bad export cookie 12116742120138914643 [ 3350.202145] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3350.204313] LustreError: 226881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3356.418412] Lustre: server umount lustre-MDT0001 complete [ 3360.839668] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 3431.570916] Lustre: server umount lustre-OST0001 complete [ 3433.729847] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 3440.574842] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3453.215756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3453.331446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3453.344649] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3453.394385] Lustre: lustre-MDT0000: new disk, initializing [ 3453.420846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3453.423943] Lustre: Skipped 3 previous similar messages [ 3453.431223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3454.862370] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3459.570672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3459.617134] Lustre: 234161:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3459.770978] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3461.251869] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3465.522271] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3467.415695] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3469.764908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3469.893589] Lustre: lustre-OST0000: new disk, initializing [ 3469.894994] Lustre: Skipped 1 previous similar message [ 3469.896991] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3469.898512] Lustre: Skipped 2 previous similar messages [ 3469.900822] Lustre: 236130:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3471.037849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3471.041052] Lustre: Skipped 1 previous similar message [ 3471.043717] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3471.082353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3472.024591] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3475.767228] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3481.571207] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3481.572658] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3481.576976] LustreError: Skipped 1 previous similar message [ 3481.586951] Lustre: Skipped 5 previous similar messages [ 3481.591622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3481.599608] Lustre: Skipped 8 previous similar messages [ 3484.466664] Lustre: server umount lustre-OST0000 complete [ 3499.999165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3500.075864] Lustre: server umount lustre-MDT0000 complete [ 3501.024267] LustreError: 234173:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3501.028785] LustreError: 234173:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3501.319748] LustreError: 234153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370049 with bad export cookie 12116742120138916113 [ 3501.323523] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3501.324432] LustreError: 234153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3501.457752] Lustre: server umount lustre-MDT0001 complete [ 3505.729509] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 3506.902696] Key type lgssc unregistered [ 3507.053606] LNet: 237891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3507.057377] LNetError: 237891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3507.067491] LNet: Removed LNI 192.168.203.123@tcp [ 3507.475143] Key type .llcrypt unregistered [ 3507.477883] Key type ._llcrypt unregistered [ 3517.465818] Key type ._llcrypt registered [ 3517.467228] Key type .llcrypt registered [ 3517.505271] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3518.042433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3518.052384] alg: No test for adler32 (adler32-zlib) [ 3518.909646] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 3519.002544] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 3520.583232] Key type lgssc registered [ 3520.949885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3523.617782] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3523.623882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3524.777397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3526.163068] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3529.053213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3529.187871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3530.538160] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3533.040507] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3535.028219] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3537.268209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3537.378942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3539.194479] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3541.504535] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3541.825755] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 3544.545534] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3544.548027] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3544.552650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3548.553532] Lustre: server umount lustre-OST0000 complete [ 3554.784752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3554.785084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3554.792392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3554.794205] Lustre: Skipped 1 previous similar message [ 3559.904121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3559.905946] Lustre: Skipped 1 previous similar message [ 3563.999197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3564.002458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3564.011314] Lustre: Skipped 1 previous similar message [ 3564.087876] Lustre: server umount lustre-MDT0000 complete [ 3565.365725] LustreError: 241166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370113 with bad export cookie 12186643500698748060 [ 3565.370516] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3565.496957] Lustre: server umount lustre-MDT0001 complete [ 3569.195594] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 3570.215029] Key type lgssc unregistered [ 3570.337370] LNet: 242907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3570.339733] LNetError: 242907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3570.347288] LNet: Removed LNI 192.168.203.123@tcp [ 3570.669855] Key type .llcrypt unregistered [ 3570.670894] Key type ._llcrypt unregistered [ 3577.407222] Key type ._llcrypt registered [ 3577.408573] Key type .llcrypt registered [ 3577.449608] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3577.991738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3578.018494] alg: No test for adler32 (adler32-zlib) [ 3578.887795] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 3578.987908] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 3580.575267] Key type lgssc registered [ 3580.938330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3583.970927] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3583.978976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3599.455326] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3604.575356] LustreError: 244073:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.123@tcp: failed processing log, type 4: rc = -110 [ 3631.199163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3633.147383] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3635.578865] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3636.314907] Lustre: server umount lustre-OST0000 complete [ 3641.787837] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 3642.799905] Key type lgssc unregistered [ 3642.926338] LNet: 245701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3642.929263] LNetError: 245701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3642.940352] LNet: Removed LNI 192.168.203.123@tcp [ 3643.258074] Key type .llcrypt unregistered [ 3643.259729] Key type ._llcrypt unregistered [ 3656.738971] Key type ._llcrypt registered [ 3656.742274] Key type .llcrypt registered [ 3656.781766] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3657.133733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3657.188580] alg: No test for adler32 (adler32-zlib) [ 3658.047708] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 3658.138655] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 3659.719205] Key type lgssc registered [ 3660.080368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3664.302785] loop0: detected capacity change from 0 to 204800000 [ 3664.362233] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3664.366535] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3664.502108] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3664.842458] loop0: detected capacity change from 0 to 204800000 [ 3664.954427] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3664.960778] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3665.088798] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3665.418174] loop0: detected capacity change from 0 to 204800000 [ 3665.476586] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3665.481145] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3665.528952] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3665.900220] loop0: detected capacity change from 0 to 204800000 [ 3665.963914] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3666.356529] loop0: detected capacity change from 0 to 204800000 [ 3666.365465] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3667.695079] loop0: detected capacity change from 0 to 204800000 [ 3667.705408] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3668.999235] loop0: detected capacity change from 0 to 204800000 [ 3669.007900] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3670.303486] loop0: detected capacity change from 0 to 204800000 [ 3670.320123] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3671.563521] loop0: detected capacity change from 0 to 204800000 [ 3671.589794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3671.599800] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3671.637623] Lustre: 249086:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3671.640965] Lustre: 249086:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 3671.644183] Lustre: 249086:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 3671.647429] Lustre: 249086:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 3671.650776] Lustre: 249086:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3671.654229] Lustre: 249086:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3671.657480] Lustre: 249086:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3671.660810] Lustre: 249086:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 3671.663964] Lustre: 249086:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 3671.667248] Lustre: 249086:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 3671.670356] Lustre: 249086:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 3671.673555] Lustre: 249086:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 3671.676884] Lustre: 249086:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 3671.680261] Lustre: 249086:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 3674.175693] Lustre: server umount MGS complete [ 3674.463383] loop0: detected capacity change from 0 to 204800000 [ 3674.497080] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3674.649981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3674.663950] Lustre: lustre-MDT0000: Aborting client recovery [ 3674.953838] loop1: detected capacity change from 0 to 204800000 [ 3674.988332] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3675.026406] Lustre: 249515:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3675.029723] Lustre: 249515:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3675.033084] Lustre: 249515:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3675.036403] Lustre: 249515:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 3675.039315] Lustre: 249515:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3675.042620] Lustre: 249515:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 3675.045665] Lustre: 249515:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 3675.052050] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3675.410034] loop2: detected capacity change from 0 to 204800000 [ 3675.438565] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3675.471305] Lustre: 249596:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3675.475732] Lustre: 249596:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3675.479217] Lustre: 249596:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3675.482832] Lustre: 249596:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3675.486184] Lustre: 249596:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3675.489645] Lustre: 249596:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3675.492827] Lustre: 249596:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3675.547478] Lustre: 249614:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3675.564502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3675.566459] Lustre: Skipped 1 previous similar message [ 3675.571406] Lustre: lustre-OST0000: Aborting recovery [ 3675.871316] loop3: detected capacity change from 0 to 204800000 [ 3675.899350] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3675.930836] Lustre: 249679:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3675.934727] Lustre: 249679:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3675.938123] Lustre: 249679:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3675.941319] Lustre: 249679:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3675.944151] Lustre: 249679:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3675.946990] Lustre: 249679:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3675.950147] Lustre: 249679:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3675.956806] Lustre: Found index 1 for lustre-OST0001, updating log [ 3675.958203] Lustre: Skipped 1 previous similar message [ 3681.252911] LustreError: 249459:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -108 [ 3684.320774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 3684.321294] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 3684.331594] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 3684.333926] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 3685.345205] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 3685.345240] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 3686.259450] LustreError: 249425:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 217us: rc = -2 [ 3700.703815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3700.704335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3700.704443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3700.707928] Lustre: Skipped 2 previous similar messages [ 3700.709531] Lustre: Skipped 1 previous similar message [ 3703.649850] Lustre: server umount lustre-MDT0000 complete [ 3703.973914] LustreError: 249407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370252 with bad export cookie 15235675649203567549 [ 3703.975667] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3703.977843] LustreError: 249407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3705.823742] LustreError: 249422:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3705.823924] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3705.824208] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3705.829494] LustreError: 249422:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3705.833681] Lustre: Skipped 2 previous similar messages [ 3708.831648] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3708.831648] LustreError: 249534:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3708.833980] Lustre: Skipped 2 previous similar messages [ 3708.841105] LustreError: 249534:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3710.255569] Lustre: server umount lustre-MDT0001 complete [ 3716.683799] Lustre: server umount lustre-OST0000 complete [ 3723.104583] Lustre: server umount lustre-OST0001 complete [ 3725.622380] Lustre: DEBUG MARKER: == conf-sanity test 108c: mkfs.lustre --mmp option enables MMP feature ========================================================== 03:17:53 (1789370273) [ 3732.531836] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3762.273545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3762.353697] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3762.365114] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3762.410175] Lustre: lustre-MDT0000: new disk, initializing [ 3762.431050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3762.433041] Lustre: Skipped 1 previous similar message [ 3762.438191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3763.781591] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3766.695165] Lustre: server umount lustre-MDT0000 complete [ 3772.342605] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 03:18:40 (1789370320) [ 3781.731869] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 3784.896813] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3796.726280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3796.804325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3796.814226] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3796.816345] Lustre: Skipped 1 previous similar message [ 3796.852127] Lustre: lustre-MDT0000: new disk, initializing [ 3796.872932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3796.878739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3798.207987] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3802.354260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3802.400495] Lustre: 256774:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3802.424610] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3802.426456] Lustre: Skipped 1 previous similar message [ 3802.463631] Lustre: lustre-MDT0001: new disk, initializing [ 3802.492777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3802.504660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3802.508552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3803.821138] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3807.444368] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3809.205510] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3811.418269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3811.515661] Lustre: lustre-OST0000: new disk, initializing [ 3811.517208] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3811.520250] Lustre: 258741:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3811.522668] Lustre: 258741:0:(osd_compat.c:1353:osd_object_spec_find()) Skipped 1 previous similar message [ 3811.548717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3813.240489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3813.243552] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3813.274581] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3813.447075] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3817.094957] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3819.386946] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 3825.041912] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 3825.044112] Lustre: Skipped 1 previous similar message [ 3874.785312] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3874.786135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3874.787671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3874.788432] LustreError: Skipped 1 previous similar message [ 3874.792206] Lustre: Skipped 1 previous similar message [ 3876.497634] Lustre: server umount lustre-OST0000 complete [ 3879.391759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3879.394132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3879.399603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3879.401297] Lustre: Skipped 1 previous similar message [ 3890.144044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3890.145978] Lustre: Skipped 4 previous similar messages [ 3892.191089] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3892.256907] Lustre: server umount lustre-MDT0000 complete [ 3893.457213] LustreError: 256767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370441 with bad export cookie 15235675649203570496 [ 3893.460478] LustreError: 256767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3893.460560] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3893.565644] Lustre: server umount lustre-MDT0001 complete [ 3896.357467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3897.671166] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3899.260083] Lustre: server umount MGS complete [ 3902.997859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3903.146965] LustreError: 262445:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3903.153989] LustreError: 262458:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 3903.176858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3904.502268] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3907.331954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3908.820464] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3911.115692] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3912.815185] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3915.304298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3917.337528] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3919.749184] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3953.632423] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3953.635616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3953.640582] Lustre: Skipped 1 previous similar message [ 3953.642718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3956.107246] Lustre: server umount lustre-OST0000 complete [ 3958.751763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3958.754016] LustreError: Skipped 1 previous similar message [ 3963.419274] Lustre: server umount lustre-MDT0000 complete [ 3964.588041] LustreError: 262427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370512 with bad export cookie 15235675649203572155 [ 3964.590805] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3964.591615] LustreError: 262427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3964.705850] Lustre: server umount lustre-MDT0001 complete [ 3968.333014] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 3969.308720] Key type lgssc unregistered [ 3969.435407] LNet: 266405:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3969.438909] LNetError: 266405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3969.448301] LNet: Removed LNI 192.168.203.123@tcp [ 3969.739870] Key type .llcrypt unregistered [ 3969.740860] Key type ._llcrypt unregistered [ 3981.051407] Key type ._llcrypt registered [ 3981.052427] Key type .llcrypt registered [ 3981.092651] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 3986.884786] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3987.186639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3987.194304] alg: No test for adler32 (adler32-zlib) [ 3988.038345] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 3988.117622] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 3989.695160] Key type lgssc registered [ 3990.024233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4001.339830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4001.344713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4002.422480] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4002.431218] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4002.466926] Lustre: lustre-MDT0000: new disk, initializing [ 4002.489120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4002.495264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4003.849822] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4007.967764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4008.008340] Lustre: 270331:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4008.030244] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4008.031862] Lustre: Skipped 1 previous similar message [ 4008.077152] Lustre: lustre-MDT0001: new disk, initializing [ 4008.108852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4008.119905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4008.123027] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4009.463232] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4013.859577] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4015.782709] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4018.259199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4018.383459] Lustre: lustre-OST0000: new disk, initializing [ 4018.385466] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4018.388522] Lustre: 272296:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4018.419941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4020.381127] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4024.261778] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4025.336945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4025.340262] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4025.373421] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4033.066248] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 4036.760786] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 4045.224491] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 4081.633260] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4081.633622] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4081.635726] LustreError: Skipped 1 previous similar message [ 4081.636337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4081.640014] Lustre: Skipped 1 previous similar message [ 4081.644599] Lustre: Skipped 1 previous similar message [ 4086.754034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4086.756103] Lustre: Skipped 1 previous similar message [ 4086.996352] Lustre: server umount lustre-OST0000 complete [ 4090.335853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4090.338111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4090.342687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4091.874469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4094.440627] Lustre: server umount lustre-MDT0000 complete [ 4095.656482] LustreError: 270322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370643 with bad export cookie 10435993434150683686 [ 4095.659161] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4095.661144] LustreError: 270322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4095.784684] Lustre: server umount lustre-MDT0001 complete [ 4098.665678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4100.112594] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4101.536286] Lustre: server umount MGS complete [ 4106.324187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4106.568898] LustreError: 276263:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 4106.612604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4109.013072] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4111.850840] LustreError: 276251:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4111.876899] LustreError: 276251:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4116.004040] LustreError: 276250:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4120.343663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4120.705140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4126.447489] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4138.104305] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4145.719900] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4153.168359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4153.548746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4161.647427] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4167.997307] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4201.953272] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4201.956870] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4201.958585] LustreError: Skipped 1 previous similar message [ 4201.959103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4201.964679] Lustre: Skipped 1 previous similar message [ 4201.967475] Lustre: Skipped 2 previous similar messages [ 4204.952973] Lustre: server umount lustre-OST0000 complete [ 4207.072722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4207.079064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4207.081792] Lustre: Skipped 1 previous similar message [ 4212.731967] Lustre: server umount lustre-MDT0000 complete [ 4214.505571] LustreError: 276230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370762 with bad export cookie 10435993434150685499 [ 4214.509259] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4214.515000] LustreError: 276230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4214.636562] Lustre: server umount lustre-MDT0001 complete [ 4219.283644] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 4220.615770] Key type lgssc unregistered [ 4220.806708] LNet: 280217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4220.809321] LNetError: 280217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4220.842699] LNet: Removed LNI 192.168.203.123@tcp [ 4221.417253] Key type .llcrypt unregistered [ 4221.419928] Key type ._llcrypt unregistered [ 4241.756991] Key type ._llcrypt registered [ 4241.765347] Key type .llcrypt registered [ 4242.118939] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 4263.956539] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4265.734979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4265.756701] alg: No test for adler32 (adler32-zlib) [ 4266.801213] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 4267.132576] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 4268.767260] Key type lgssc registered [ 4270.267200] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4298.196866] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4312.589793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4312.633428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4314.020259] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4314.065430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4314.206546] Lustre: lustre-MDT0000: new disk, initializing [ 4314.286996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4314.316601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4319.180094] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4328.760972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4329.311699] Lustre: lustre-OST0000: new disk, initializing [ 4329.315329] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4329.320218] Lustre: Skipped 1 previous similar message [ 4329.324960] Lustre: 284293:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4329.395236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4334.672095] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4339.290884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 4339.303867] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 4339.336807] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 5137.891175] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5137.919292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5140.148293] Lustre: server umount lustre-MDT0000 complete [ 5143.200921] LustreError: 284292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789371691 with bad export cookie 5811081844592638086 [ 5143.203097] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5143.210965] LustreError: 284292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5143.315385] Lustre: server umount lustre-OST0000 complete [ 5166.694694] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 5181.960546] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5222.361740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5222.633249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5222.707524] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5222.802789] Lustre: lustre-MDT0000: new disk, initializing [ 5223.013967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5223.046085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5227.393586] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5237.542510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5237.659862] Lustre: 288647:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5237.729529] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5237.736423] Lustre: Skipped 1 previous similar message [ 5237.838993] Lustre: lustre-MDT0001: new disk, initializing [ 5238.000826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5238.054092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5238.063073] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5242.684035] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5251.572891] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5256.678364] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5262.595089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5262.844277] Lustre: lustre-OST0000: new disk, initializing [ 5262.847892] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5262.853151] Lustre: 290613:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5262.947111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5267.974706] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5271.628067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5271.645056] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5271.760748] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5276.098594] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5280.971052] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5281.140738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5285.468673] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5285.662572] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5287.392121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5287.395222] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5287.407329] LustreError: Skipped 1 previous similar message [ 5287.419666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5292.519341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5292.522887] Lustre: Skipped 2 previous similar messages [ 5293.557965] Lustre: server umount lustre-OST0000 complete [ 5297.634065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5297.647635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5297.659511] Lustre: Skipped 2 previous similar messages [ 5302.757610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5302.768884] Lustre: Skipped 2 previous similar messages [ 5303.244458] Lustre: server umount lustre-MDT0000 complete [ 5306.661262] LustreError: 288640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789371854 with bad export cookie 5811081844595578905 [ 5306.672043] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5306.674596] LustreError: 288640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5306.965628] Lustre: server umount lustre-MDT0001 complete [ 5309.976333] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [ 5311.372492] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 03:44:18 (1789371858) [ 5316.834048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5317.388252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5321.600842] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5322.721638] LustreError: 292928:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5322.746498] LustreError: 292928:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5327.841929] LustreError: 292927:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5328.690527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5329.049117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5332.802953] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5339.152379] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5344.150482] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5349.859347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5350.218714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5355.985891] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5362.970909] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5369.517497] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5369.737897] Lustre: lustre-OST0001: new disk, initializing [ 5369.745864] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5369.753814] Lustre: 295762:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5369.875464] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5376.373734] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5377.136807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5377.148845] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5377.222219] LustreError: 296151:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 5378.281096] LustreError: 296157:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 5378.297498] LustreError: 296157:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 5379.297755] LustreError: 296151:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 5381.377138] LustreError: 296151:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 5381.390980] LustreError: 296151:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 5385.512507] LustreError: 296151:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 5385.525043] LustreError: 296151:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [ 5386.368957] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5386.663201] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5392.227462] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9302048e9800.ost_server_uuid 50 [ 5393.596971] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9302048e9800.ost_server_uuid in FULL state after 0 sec [ 5393.706491] LustreError: 296157:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 5393.719571] LustreError: 296157:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 14 previous similar messages [ 5398.640397] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9302048e9800.ost_server_uuid 50 [ 5400.043837] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9302048e9800.ost_server_uuid in FULL state after 0 sec [ 5401.978244] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5417.405838] Lustre: lustre-OST0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 5417.442297] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5417.447700] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5417.479951] LustreError: Skipped 2 previous similar messages [ 5429.215331] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5429.340812] Lustre: server umount lustre-OST0001 complete [ 5432.709226] LustreError: 294749:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5436.384683] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5436.391482] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5436.407775] Lustre: Skipped 1 previous similar message [ 5436.413490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5436.415791] Lustre: Skipped 10 previous similar messages [ 5438.958214] LustreError: 295378:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5438.976018] LustreError: 295378:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5441.538218] Lustre: server umount lustre-OST0000 complete [ 5446.625466] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5446.636139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5446.648301] Lustre: Skipped 1 previous similar message [ 5451.474684] Lustre: server umount lustre-MDT0000 complete [ 5454.305916] LustreError: 292928:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5454.332486] LustreError: 292928:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5454.459487] LustreError: 294769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789372002 with bad export cookie 5811081844595579787 [ 5454.460599] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5454.472215] LustreError: 294769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5454.753841] Lustre: server umount lustre-MDT0001 complete [ 5464.046283] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5466.470674] Key type lgssc unregistered [ 5466.754847] LNet: 298450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5466.761775] LNetError: 298450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5466.783247] LNet: Removed LNI 192.168.203.123@tcp [ 5467.561802] Key type .llcrypt unregistered [ 5467.563834] Key type ._llcrypt unregistered [ 5491.810294] Key type ._llcrypt registered [ 5491.816965] Key type .llcrypt registered [ 5491.913604] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5493.024715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5493.103597] alg: No test for adler32 (adler32-zlib) [ 5494.258152] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 5494.602272] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 5496.343200] Key type lgssc registered [ 5497.714945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5505.070367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5505.090278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5506.516104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5511.000699] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5516.769541] LustreError: 299867:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5516.793333] LustreError: 299867:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5518.181418] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5521.890366] LustreError: 299868:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5523.850466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5524.195653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5528.731249] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5538.825974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5539.116183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5547.559144] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5549.369886] LustreError: 301547:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5551.473586] LustreError: 301548:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5551.516883] LustreError: 301548:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5551.534373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 5556.584327] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5556.717425] LustreError: 301549:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5556.744849] LustreError: 301549:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5564.173751] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5564.539044] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5569.520349] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5571.784290] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5578.365557] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5584.825566] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5585.052514] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5687.267413] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5687.271359] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5687.282924] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5690.276633] Lustre: server umount lustre-OST0001 complete [ 5692.392281] LustreError: 301548:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5692.406234] LustreError: 301548:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5697.507399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5697.516720] Lustre: Skipped 1 previous similar message [ 5697.522268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5697.526600] Lustre: Skipped 2 previous similar messages [ 5702.632932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5702.644163] Lustre: Skipped 1 previous similar message [ 5707.749239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5707.764940] Lustre: Skipped 1 previous similar message [ 5710.303261] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5710.435862] Lustre: server umount lustre-OST0000 complete [ 5713.890187] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5713.898125] LustreError: Skipped 1 previous similar message [ 5713.902837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5713.916873] Lustre: Skipped 1 previous similar message [ 5713.921656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5717.986032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5719.805755] Lustre: server umount lustre-MDT0000 complete [ 5723.080042] LustreError: 299849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789372271 with bad export cookie 13885699259563501697 [ 5723.082994] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5723.086829] LustreError: 299849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5723.366859] Lustre: server umount lustre-MDT0001 complete [ 5734.309258] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5736.847695] Key type lgssc unregistered [ 5737.162206] LNet: 304811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5737.169993] LNetError: 304811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5737.187742] LNet: Removed LNI 192.168.203.123@tcp [ 5738.448993] Key type .llcrypt unregistered [ 5738.452437] Key type ._llcrypt unregistered [ 5780.774628] Key type ._llcrypt registered [ 5780.777500] Key type .llcrypt registered [ 5780.854353] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5781.858375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5781.880223] alg: No test for adler32 (adler32-zlib) [ 5782.970214] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 5783.261119] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 5784.911212] Key type lgssc registered [ 5786.116574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5793.976560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5793.988543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5795.485175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5799.817391] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5805.027475] LustreError: 307097:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5805.039582] LustreError: 307097:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5807.857706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5808.225874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5811.812682] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5813.984295] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5819.921259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5820.396626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5821.413421] LustreError: 308586:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5821.427155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 5826.260201] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5826.541978] LustreError: 308586:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5831.655095] LustreError: 308589:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5834.206949] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5834.475770] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5839.873630] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 5840.897501] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5849.616757] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5858.439763] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5870.570725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5870.579580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5870.583519] Lustre: Skipped 2 previous similar messages [ 5870.599912] Lustre: Skipped 3 previous similar messages [ 5875.682107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5875.697086] Lustre: Skipped 3 previous similar messages [ 5876.080686] Lustre: server umount lustre-MDT0000 complete [ 5879.459627] LustreError: 307076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789372427 with bad export cookie 436399182736154241 [ 5879.467961] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5879.474528] LustreError: 307076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5879.746206] Lustre: server umount lustre-MDT0001 complete [ 5893.600603] Lustre: server umount lustre-OST0000 complete [ 5907.012431] Lustre: server umount lustre-OST0001 complete [ 5923.936057] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5933.469639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5933.957107] LustreError: 312361:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5933.981381] LustreError: 312361:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5934.056957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5938.283522] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5944.289575] LustreError: 312362:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5944.310192] LustreError: 312362:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5946.473553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5946.947472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5951.123991] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5953.680848] Lustre: 313501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5959.897743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5966.171730] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5971.697251] LustreError: 313854:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5974.317486] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5974.627459] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5974.634419] Lustre: Skipped 1 previous similar message [ 5980.681254] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 5980.696129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5980.729609] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5987.791026] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5991.250810] Lustre: 315370:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5996.002436] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5996.010591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5996.012770] Lustre: Skipped 4 previous similar messages [ 5996.031121] Lustre: Skipped 3 previous similar messages [ 6001.188435] Lustre: server umount lustre-MDT0000 complete [ 6004.251254] LustreError: 312341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789372552 with bad export cookie 436399182736155942 [ 6004.253657] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6004.261928] LustreError: 312341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6004.596153] Lustre: server umount lustre-MDT0001 complete [ 6018.480828] Lustre: server umount lustre-OST0000 complete [ 6031.295284] Lustre: server umount lustre-OST0001 complete [ 6035.034686] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [ 6036.486574] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [ 6038.383247] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 03:56:25 (1789372585) [ 6059.775458] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6068.305262] loop0: detected capacity change from 0 to 1099511627776 [ 6068.971950] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 6068.990041] *** VALIDATE xfs *** [ 6069.005456] XFS (loop0): Mounting V5 Filesystem [ 6069.026574] XFS (loop0): Ending clean mount [ 6070.297731] loop1: detected capacity change from 0 to 18691697672192 [ 6173.886904] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 6187.189841] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6227.376142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6227.703029] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6227.743798] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6227.805678] Lustre: lustre-MDT0000: new disk, initializing [ 6227.894142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6227.932045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6232.540400] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6242.805078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6242.920213] Lustre: 320511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6243.014603] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6243.020556] Lustre: Skipped 1 previous similar message [ 6243.118475] Lustre: lustre-MDT0001: new disk, initializing [ 6243.255382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6243.266805] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6247.239478] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6255.301620] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6260.105662] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6265.604968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6265.831913] Lustre: lustre-OST0000: new disk, initializing [ 6265.837715] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6265.842668] Lustre: 322476:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6267.139682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6267.151557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6267.310165] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6271.619852] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6281.711558] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6286.779459] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6287.067196] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6291.971409] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6292.183037] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6298.081241] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6298.088997] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6298.095853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6298.110796] Lustre: Skipped 1 previous similar message [ 6303.202040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6303.210250] Lustre: Skipped 2 previous similar messages [ 6308.319181] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6308.458508] Lustre: server umount lustre-OST0000 complete [ 6313.445721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6313.456989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6313.474810] Lustre: Skipped 1 previous similar message [ 6313.496859] Lustre: Skipped 3 previous similar messages [ 6318.562642] LustreError: 321376:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6318.574131] LustreError: 321376:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6318.621738] Lustre: server umount lustre-MDT0000 complete [ 6321.677119] LustreError: 320501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789372869 with bad export cookie 436399182736157496 [ 6321.684689] LustreError: 320501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6321.692054] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6321.968615] Lustre: server umount lustre-MDT0001 complete [ 6329.338994] XFS (loop0): Unmounting Filesystem [ 6334.446383] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 04:01:21 (1789372881) [ 6340.529459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6340.964165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6340.968619] Lustre: Skipped 2 previous similar messages [ 6344.924741] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6352.925634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6357.304300] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6363.956509] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6368.624165] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6374.758372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6380.903629] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6386.505357] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6393.827894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6393.835482] LustreError: Skipped 1 previous similar message [ 6393.837126] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6393.842884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6396.905305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6398.193272] Lustre: server umount lustre-OST0000 complete [ 6402.021381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6402.034693] Lustre: Skipped 2 previous similar messages [ 6416.357307] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6416.537936] Lustre: server umount lustre-MDT0000 complete [ 6417.378055] LustreError: 326526:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6417.386046] LustreError: 326526:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6420.798370] LustreError: 325029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789372968 with bad export cookie 436399182736158378 [ 6420.802662] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6420.805276] LustreError: 325029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6421.453213] Lustre: server umount lustre-MDT0001 complete [ 6432.954264] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 6436.322547] Key type lgssc unregistered [ 6436.632774] LNet: 328678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6436.639344] LNetError: 328678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6436.655590] LNet: Removed LNI 192.168.203.123@tcp [ 6437.613145] Key type .llcrypt unregistered [ 6437.614852] Key type ._llcrypt unregistered [ 6462.643598] Key type ._llcrypt registered [ 6462.646835] Key type .llcrypt registered [ 6462.746770] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6463.626655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6463.769289] alg: No test for adler32 (adler32-zlib) [ 6464.780790] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 6465.093117] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 6466.743205] Key type lgssc registered [ 6467.870918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6474.716365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6474.736442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6476.336320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6480.676559] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6486.498377] LustreError: 330242:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6486.534271] LustreError: 330242:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6489.034873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6489.380575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6493.798616] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6501.567248] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6507.150446] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6513.606936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6513.986090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6519.573932] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6527.020524] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6535.076228] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 6535.649308] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6535.657755] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6535.679144] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6536.682244] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6536.698946] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6536.707748] Lustre: Skipped 1 previous similar message [ 6540.167812] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 6540.869585] Lustre: server umount lustre-MDT0001 complete [ 6541.800090] LustreError: 332155:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6541.823302] LustreError: 332155:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6545.293262] LustreError: 332736:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6550.407548] LustreError: 330238:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6550.439596] LustreError: 330238:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6551.666739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6551.824690] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6551.833845] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6551.914656] Lustre: 330229:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6552.043721] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6552.287041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6557.364469] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6557.683939] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6561.785932] LustreError: 333845:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 6561.796442] LustreError: 333845:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 6561.804933] LustreError: 330234:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 6561.937858] LustreError: 333848:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.123@tcp: cfg command failed: rc = -17 [ 6561.966210] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 6561.966210] [ 6864.865042] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6864.872489] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6864.881178] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6864.885643] Lustre: Skipped 1 previous similar message [ 6864.913889] Lustre: Skipped 2 previous similar messages [ 6870.007746] LustreError: 330237:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6870.057411] LustreError: 330237:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6870.413431] Lustre: server umount lustre-MDT0001 complete [ 6878.088153] LustreError: 332736:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6878.120711] LustreError: 332736:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6884.513930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6884.738591] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6884.761311] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6884.882438] Lustre: 330229:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6885.362851] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 6889.300433] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6890.478605] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6892.017800] LustreError: 342818:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 6892.024323] LustreError: 342818:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 6892.032153] LustreError: 330234:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 6892.035419] LustreError: 330234:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 6892.088393] LustreError: 342821:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [ 6892.097155] LustreError: 342821:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [ 6892.105433] LustreError: 342821:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.123@tcp: cfg command failed: rc = -17 [ 6892.114191] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.203.123@tcp [ 6892.114191] [ 7197.683041] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7197.690990] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7197.710504] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7197.715154] Lustre: Skipped 1 previous similar message [ 7201.369777] Lustre: server umount lustre-MDT0001 complete [ 7202.790753] LustreError: 330242:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7202.805305] LustreError: 330242:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7210.987437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7211.073675] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7211.077555] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7211.145582] Lustre: 330232:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7211.508935] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 7215.831725] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7216.621112] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7218.193566] LustreError: 351902:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 7218.207543] LustreError: 351902:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 7218.226979] LustreError: 330234:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 7218.236430] LustreError: 330234:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 7218.354707] LustreError: 351911:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 7218.380615] LustreError: 351911:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.123@tcp: cfg command failed: rc = -114 [ 7218.390632] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 7218.390632] [ 7539.172949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7539.186024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7539.229480] Lustre: Skipped 3 previous similar messages [ 7539.246403] Lustre: Skipped 3 previous similar messages [ 7542.077301] Lustre: server umount lustre-MDT0000 complete [ 7544.294362] LustreError: 330237:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7544.320678] LustreError: 330237:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 7545.913095] LustreError: 330223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789374094 with bad export cookie 10790137380084041873 [ 7545.928382] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7549.408961] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7552.236937] Lustre: server umount lustre-MDT0001 complete [ 7556.314490] Lustre: server umount lustre-OST0000 complete [ 7566.224400] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 7577.008398] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7618.065058] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 04:22:44 (1789374164) [ 7626.194576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7626.507861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7626.552297] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7626.637819] Lustre: lustre-MDT0000: new disk, initializing [ 7626.723387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7626.744191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7630.562855] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7642.290026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7642.458834] Lustre: 362776:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7642.521423] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7642.526437] Lustre: Skipped 1 previous similar message [ 7642.609211] Lustre: lustre-MDT0001: new disk, initializing [ 7642.757716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7642.825679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7642.833866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7647.152761] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7656.236697] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7664.379697] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7674.617466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7675.169929] Lustre: lustre-OST0000: new disk, initializing [ 7675.186757] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7675.203797] Lustre: 364744:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7675.344516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7682.166550] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7683.614215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7683.624612] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7683.743457] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7692.128779] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7700.971104] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7700.989650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7701.033415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7701.043471] Lustre: Skipped 2 previous similar messages [ 7701.834275] Lustre: server umount lustre-MDT0000 complete [ 7704.047463] LustreError: 362787:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7704.072334] LustreError: 362787:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7705.731365] LustreError: 362767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789374253 with bad export cookie 10790137380084043819 [ 7705.749379] LustreError: 362767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7705.757413] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7706.414882] Lustre: server umount lustre-MDT0001 complete [ 7720.631702] Lustre: server umount lustre-OST0000 complete [ 7749.057790] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 7762.511586] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7800.990660] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7578 sec ========== 04:25:47 (1789374347) [ 7802.557980] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:25:49 (1789374349) === [ 7805.901819] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:25:52 (1789374352) === [ 7850.766438] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7853.536306] Key type lgssc unregistered [ 7853.838750] LNet: 371112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7853.845402] LNetError: 371112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7853.859415] LNet: Removed LNI 192.168.203.123@tcp [ 7855.394846] Key type .llcrypt unregistered [ 7855.404208] Key type ._llcrypt unregistered