[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 543871114 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003247] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008299] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011138] LSM: Security Framework initializing [ 0.012059] Yama: becoming mindful. [ 0.013034] SELinux: Initializing. [ 0.015067] *** VALIDATE selinux *** [ 0.023512] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027808] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028185] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029125] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030117] *** VALIDATE tmpfs *** [ 0.031459] *** VALIDATE proc *** [ 0.033196] *** VALIDATE cgroup *** [ 0.034012] *** VALIDATE cgroup2 *** [ 0.036040] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038087] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040032] Spectre V2 : User space: Vulnerable [ 0.041009] Speculative Store Bypass: Vulnerable [ 0.044197] debug: unmapping init [mem 0xffffffffb5859000-0xffffffffb5860fff] [ 0.046131] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047661] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048017] ... version: 2 [ 0.048990] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.049757] ... value mask: 0000ffffffffffff [ 0.050009] ... max period: 00007fffffffffff [ 0.051011] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.053289] rcu: Hierarchical SRCU implementation. [ 0.055528] smp: Bringing up secondary CPUs ... [ 0.056607] x86: Booting SMP configuration: [ 0.057026] .... node #0, CPUs: #1 #2 #3 [ 0.060813] smp: Brought up 1 node, 4 CPUs [ 0.063015] smpboot: Max logical packages: 1 [ 0.064019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.218248] node 0 deferred pages initialised in 151ms [ 0.221120] devtmpfs: initialized [ 0.223351] x86/mm: Memory block size: 128MB [ 0.227213] gcov: version magic: 0x41383552 [ 0.233109] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.234066] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.236362] pinctrl core: initialized pinctrl subsystem [ 0.238274] [ 0.238913] ************************************************************* [ 0.241017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.243012] ** ** [ 0.246014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.248010] ** ** [ 0.250013] ** This means that this kernel is built to expose internal ** [ 0.252013] ** IOMMU data structures, which may compromise security on ** [ 0.254017] ** your system. ** [ 0.256015] ** ** [ 0.258014] ** If you see this message and you are not debugging the ** [ 0.261016] ** kernel, report this immediately to your vendor! ** [ 0.263012] ** ** [ 0.265027] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.268012] ************************************************************* [ 0.270923] NET: Registered protocol family 16 [ 0.274552] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.276070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.279073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.282052] cpuidle: using governor menu [ 0.285990] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.290454] PCI: Using configuration type 1 for base access [ 0.293134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.304067] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.305043] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.307751] cryptd: max_cpu_qlen set to 1000 [ 0.310227] ACPI: Added _OSI(Module Device) [ 0.312022] ACPI: Added _OSI(Processor Device) [ 0.313019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.315016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.320961] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.328678] ACPI: Interpreter enabled [ 0.330075] ACPI: PM: (supports S0 S3 S4 S5) [ 0.333023] ACPI: Using IOAPIC for interrupt routing [ 0.335226] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.340462] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.354858] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.357052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.360032] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.365119] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.372510] acpiphp: Slot [2] registered [ 0.374170] acpiphp: Slot [5] registered [ 0.376236] acpiphp: Slot [6] registered [ 0.378237] acpiphp: Slot [7] registered [ 0.380151] acpiphp: Slot [8] registered [ 0.382211] acpiphp: Slot [9] registered [ 0.384179] acpiphp: Slot [10] registered [ 0.387174] acpiphp: Slot [3] registered [ 0.389170] acpiphp: Slot [4] registered [ 0.391150] acpiphp: Slot [11] registered [ 0.393119] acpiphp: Slot [12] registered [ 0.395128] acpiphp: Slot [13] registered [ 0.397165] acpiphp: Slot [14] registered [ 0.399147] acpiphp: Slot [15] registered [ 0.401211] acpiphp: Slot [16] registered [ 0.404210] acpiphp: Slot [17] registered [ 0.406284] acpiphp: Slot [18] registered [ 0.408134] acpiphp: Slot [19] registered [ 0.410146] acpiphp: Slot [20] registered [ 0.412112] acpiphp: Slot [21] registered [ 0.414142] acpiphp: Slot [22] registered [ 0.416114] acpiphp: Slot [23] registered [ 0.417258] acpiphp: Slot [24] registered [ 0.419198] acpiphp: Slot [25] registered [ 0.420100] acpiphp: Slot [26] registered [ 0.421000] acpiphp: Slot [27] registered [ 0.422053] acpiphp: Slot [28] registered [ 0.424116] acpiphp: Slot [29] registered [ 0.426123] acpiphp: Slot [30] registered [ 0.427178] acpiphp: Slot [31] registered [ 0.429071] PCI host bridge to bus 0000:00 [ 0.431038] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.434036] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.438048] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.441026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.445030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.448025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.451215] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.455078] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.459675] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.469884] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.475059] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.478019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.480018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.484602] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.487665] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.490778] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.494060] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.498925] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.505071] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.522026] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.528022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.535305] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.545017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.554023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.581019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.597948] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.620018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.636015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.669019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.691627] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.730028] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.738025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.754019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.767800] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.778029] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.790065] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.822021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.843624] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.856070] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.868033] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.891026] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.906220] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.918019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.925015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.943017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.975882] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.977255] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.979263] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.981231] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.985190] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.989146] iommu: Default domain type: Passthrough [ 0.992494] SCSI subsystem initialized [ 0.995175] ACPI: bus type USB registered [ 0.997111] usbcore: registered new interface driver usbfs [ 1.000130] usbcore: registered new interface driver hub [ 1.003103] usbcore: registered new device driver usb [ 1.005171] pps_core: LinuxPPS API ver. 1 registered [ 1.008015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.012056] PTP clock support registered [ 1.015160] EDAC MC: Ver: 3.0.0 [ 1.017539] PCI: Using ACPI for IRQ routing [ 1.020768] NetLabel: Initializing [ 1.022010] NetLabel: domain hash size = 128 [ 1.024016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.028088] NetLabel: unlabeled traffic allowed by default [ 1.030139] vgaarb: loaded [ 1.033060] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.036025] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.045935] clocksource: Switched to clocksource kvm-clock [ 1.174531] VFS: Disk quotas dquot_6.6.0 [ 1.177233] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.181219] *** VALIDATE ramfs *** [ 1.182720] *** VALIDATE hugetlbfs *** [ 1.185791] pnp: PnP ACPI init [ 1.190406] pnp: PnP ACPI: found 6 devices [ 1.210683] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.215964] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.219343] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.222653] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.225653] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.228253] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.231168] NET: Registered protocol family 2 [ 1.233749] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.239777] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.244721] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.252373] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.257839] TCP: Hash tables configured (established 65536 bind 65536) [ 1.262659] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.266711] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.270480] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.274494] NET: Registered protocol family 1 [ 1.278456] RPC: Registered named UNIX socket transport module. [ 1.281958] RPC: Registered udp transport module. [ 1.284638] RPC: Registered tcp transport module. [ 1.287657] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.291459] NET: Registered protocol family 44 [ 1.293086] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.295893] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.298471] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.301161] PCI: CLS 0 bytes, default 64 [ 1.303174] Unpacking initramfs... [ 3.111488] hrtimer: interrupt took 4172742 ns [ 4.922801] debug: unmapping init [mem 0xffff8a8dbcc54000-0xffff8a8dbffbffff] [ 4.931231] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.936459] software IO TLB: mapped [mem 0x00000000a2800000-0x00000000a6800000] (64MB) [ 4.947410] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.814622] Initialise system trusted keyrings [ 6.817581] Key type blacklist registered [ 6.822910] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.850206] zbud: loaded [ 6.860119] *** VALIDATE nfs *** [ 6.862293] *** VALIDATE nfs4 *** [ 6.874564] pstore: using deflate compression [ 6.888425] Platform Keyring initialized [ 7.357441] NET: Registered protocol family 38 [ 7.359566] Key type asymmetric registered [ 7.361351] Asymmetric key parser 'x509' registered [ 7.363656] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 7.369698] io scheduler mq-deadline registered [ 7.372287] io scheduler kyber registered [ 7.373371] io scheduler bfq registered [ 7.376393] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 7.384434] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 7.389432] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 7.398648] ACPI: Power Button [PWRF] [ 7.410171] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.421442] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.442612] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 7.456239] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 7.500075] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.535563] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.591799] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.646651] Non-volatile memory driver v1.3 [ 7.651273] Linux agpgart interface v0.103 [ 7.761579] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 7.771255] vda: detected capacity change from 0 to 74702848 [ 7.818357] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.822728] vdb: detected capacity change from 0 to 1073741824 [ 7.896241] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.914230] vdc: detected capacity change from 0 to 2621440000 [ 7.966662] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.972533] vdd: detected capacity change from 0 to 2621440000 [ 8.013180] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.022769] vde: detected capacity change from 0 to 4294967296 [ 8.069142] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.075293] vdf: detected capacity change from 0 to 4294967296 [ 8.094103] libphy: Fixed MDIO Bus: probed [ 8.142626] usbcore: registered new interface driver usbserial_generic [ 8.144847] usbserial: USB Serial support registered for generic [ 8.149572] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 8.163131] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 8.165662] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 8.170455] mousedev: PS/2 mouse device common for all mice [ 8.174555] rtc_cmos 00:05: RTC can wake from S4 [ 8.179317] rtc_cmos 00:05: registered as rtc0 [ 8.182413] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 8.187511] intel_pstate: CPU model not supported [ 8.187802] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 8.193274] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 8.200433] hid: raw HID events driver (C) Jiri Kosina [ 8.210446] usbcore: registered new interface driver usbhid [ 8.213647] usbhid: USB HID core driver [ 8.216161] drop_monitor: Initializing network drop monitor service [ 8.219991] Initializing XFRM netlink socket [ 8.220031] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 8.222586] NET: Registered protocol family 10 [ 8.223853] Segment Routing with IPv6 [ 8.223888] NET: Registered protocol family 17 [ 8.224457] mpls_gso: MPLS GSO support [ 8.238060] RAS: Correctable Errors collector initialized. [ 8.240975] AVX version of gcm_enc/dec engaged. [ 8.243451] AES CTR mode by8 optimization enabled [ 8.386184] sched_clock: Marking stable (8386164621, 0)->(9265189874, -879025253) [ 8.392316] registered taskstats version 1 [ 8.395699] Loading compiled-in X.509 certificates [ 8.400230] zswap: loaded using pool lzo/zbud [ 8.448397] Key type big_key registered [ 8.479560] Key type encrypted registered [ 8.482321] ima: No TPM chip found, activating TPM-bypass! [ 8.486652] ima: Allocated hash algorithm: sha1 [ 8.489716] ima: No architecture policies found [ 8.494169] evm: Initialising EVM extended attributes: [ 8.495957] evm: security.selinux [ 8.498485] evm: security.ima [ 8.499891] evm: security.capability [ 8.501447] evm: HMAC attrs: 0x1 [ 8.505566] rtc_cmos 00:05: setting system clock to 2026-08-13 13:18:27 UTC (1786627107) [ 8.518151] debug: unmapping init [mem 0xffffffffb6803000-0xffffffffb69fffff] [ 8.523951] debug: unmapping init [mem 0xffffffffb5582000-0xffffffffb5858fff] [ 8.534495] Write protecting the kernel read-only data: 28672k [ 8.540189] debug: unmapping init [mem 0xffffffffb3c03000-0xffffffffb3dfffff] [ 8.546561] debug: unmapping init [mem 0xffffffffb4514000-0xffffffffb45fffff] [ 8.605749] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.619502] systemd[1]: Detected virtualization kvm. [ 8.625995] systemd[1]: Detected architecture x86-64. [ 8.634761] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.691463] systemd[1]: No hostname configured. [ 8.694149] systemd[1]: Set hostname to . [ 8.701052] random: systemd: uninitialized urandom read (16 bytes read) [ 8.704428] systemd[1]: Initializing machine ID from random generator. [ 8.849300] random: ln: uninitialized urandom read (6 bytes read) [ 8.989230] random: systemd: uninitialized urandom read (16 bytes read) [ 8.993924] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 9.000569] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 9.013395] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Reached target Swap. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.474453] device-mapper: uevent: version 1.0.3 [ 10.478363] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook...[ 12.315906] random: fast init done [ OK ] Reached target System Initialization. [ 12.342811] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 12.649288] scsi host0: ata_piix [ 12.670497] scsi host1: ata_piix [ 12.672052] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.680685] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 18.026273] random: crng init done [ 18.035625] random: 7 urandom warning(s) missed due to ratelimiting [ 19.105882] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 20.698186] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.975261] printk: systemd: 26 output lines suppressed due to ratelimiting [ 25.004807] SELinux: Disabled at runtime. [ 25.129191] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.141355] systemd[1]: Detected virtualization kvm. [ 25.142632] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 27.140381] systemd[1]: initrd-switch-root.service: Succeeded. [ 27.147740] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 27.168257] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 27.179235] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 27.194193] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 27.235179] systemd[1]: Starting Journal Service... Starting Journal Service... [ 27.258329] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ 27.482169] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Paths. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... [ OK ] Reached target Swap. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 29.707540] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 31.154447] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 31.516307] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 32.438971] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 32.636846] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [[ 37.776797] Key type dns_resolver registered *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit)[ 38.274474] NFS: Registering the id_resolver key type [ 38.275912] Key type id_resolver registered [ 38.277158] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg117-server login: [ 100.974538] libcfs: loading out-of-tree module taints kernel. [ 101.008175] Key type ._llcrypt registered [ 101.009741] Key type .llcrypt registered [ 101.064867] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 121.710492] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 123.098852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 123.107603] alg: No test for adler32 (adler32-zlib) [ 124.947366] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 125.968241] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 127.890935] Key type lgssc registered [ 130.136824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 150.696337] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 194.935936] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 208.252862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 208.310357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.626605] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 209.703443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 209.859324] Lustre: lustre-MDT0000: new disk, initializing [ 209.972464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 209.989425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 213.672874] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.818329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 226.918357] Lustre: 6521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 226.961980] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 226.967927] Lustre: Skipped 1 previous similar message [ 227.074935] Lustre: lustre-MDT0001: new disk, initializing [ 227.153849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 227.193923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 227.207983] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 232.078674] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.183405] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 249.146971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 249.459962] Lustre: lustre-OST0000: new disk, initializing [ 249.467242] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 249.477815] Lustre: 8463:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 249.569382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 255.594626] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 258.576419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 258.589917] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 258.710917] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 270.825163] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 270.943603] Lustre: lustre-OST0001: new disk, initializing [ 270.947447] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 270.952529] Lustre: 9535:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 271.015563] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 277.059101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 277.082355] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 277.129331] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 278.235730] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 291.271690] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 298.601212] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 304.954800] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing check_logdir /tmp/testlogs/ [ 313.257065] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing yml_node [ 317.613797] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 320.310853] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 323.503369] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 325.747585] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 13 09:23:43 EDT 2026 [ 345.837209] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 347.477334] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 364.004588] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.006458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 364.026881] Lustre: Skipped 3 previous similar messages [ 364.038264] Lustre: Skipped 2 previous similar messages [ 369.121038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 369.130585] Lustre: Skipped 3 previous similar messages [ 370.027591] Lustre: server umount lustre-MDT0000 complete [ 374.473991] LustreError: 8462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786627473 with bad export cookie 6091054840282800289 [ 374.480571] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 374.501355] LustreError: 8462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 379.370126] LustreError: 7378:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 379.373532] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 379.379987] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 379.393122] LustreError: 7378:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 379.420599] Lustre: Skipped 1 previous similar message [ 380.927940] Lustre: server umount lustre-MDT0001 complete [ 385.955817] Lustre: server umount lustre-OST0000 complete [ 390.083733] Lustre: server umount lustre-OST0001 complete [ 398.173762] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 410.448964] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 449.212472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 449.550758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 449.596353] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 449.748489] Lustre: lustre-MDT0000: new disk, initializing [ 449.900629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 449.933056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 454.388841] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 466.154389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 466.266780] Lustre: 15092:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 466.361516] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 466.371369] Lustre: Skipped 1 previous similar message [ 466.481478] Lustre: lustre-MDT0001: new disk, initializing [ 466.617564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 466.662784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 466.680449] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 471.601907] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 481.909573] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 488.434747] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 496.045509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 496.460199] Lustre: lustre-OST0000: new disk, initializing [ 496.467647] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 496.479940] Lustre: 17060:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 496.595633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 498.413322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 498.427312] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 498.605911] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 505.085915] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 517.124337] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 523.862619] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 524.133532] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 530.745846] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 531.029484] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 534.499245] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 534.499638] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 534.510785] LustreError: Skipped 1 previous similar message [ 534.527678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 534.539806] Lustre: Skipped 2 previous similar messages [ 539.426873] Lustre: server umount lustre-OST0000 complete [ 543.713523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 543.720210] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 543.739186] Lustre: Skipped 1 previous similar message [ 543.751187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 549.862259] LustreError: 15103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.886970] LustreError: 15103:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 550.018050] Lustre: server umount lustre-MDT0000 complete [ 554.986183] LustreError: 16223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 555.017965] LustreError: 16223:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 555.209892] LustreError: 15083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786627654 with bad export cookie 6091054840282801822 [ 555.215238] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 555.225103] LustreError: 15083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 555.812718] Lustre: server umount lustre-MDT0001 complete [ 562.492656] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 09:27:40 (1786627660) [ 571.655142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 572.518866] LustreError: 19328:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.603577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 577.691787] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 578.030061] LustreError: 19329:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 587.235142] LustreError: 19328:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 587.251634] LustreError: 19328:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 587.718401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 593.488420] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 601.131487] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 606.384585] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 613.717731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 620.979321] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 629.058207] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 636.210944] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 644.071413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 644.081451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 644.110864] Lustre: Skipped 3 previous similar messages [ 644.123928] Lustre: Skipped 3 previous similar messages [ 649.852804] Lustre: server umount lustre-MDT0000 complete [ 653.652969] LustreError: 21858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786627752 with bad export cookie 6091054840282802704 [ 653.668340] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 653.674687] LustreError: 21858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 654.016606] Lustre: server umount lustre-MDT0001 complete [ 669.770813] Lustre: server umount lustre-OST0000 complete [ 690.334345] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 700.404807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 700.737503] LustreError: 23710:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 700.796946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 700.801667] Lustre: Skipped 2 previous similar messages [ 706.315736] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 716.462257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 722.501864] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 725.615031] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 731.352850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 739.427187] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 748.079826] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 748.266189] Lustre: lustre-OST0001: new disk, initializing [ 748.272388] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 748.278471] Lustre: 26029:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 750.288450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 750.298111] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 750.400513] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 755.004439] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 764.947973] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 769.572067] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 781.291513] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 781.297211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 781.298019] Lustre: Skipped 2 previous similar messages [ 781.303199] Lustre: Skipped 7 previous similar messages [ 787.529767] Lustre: server umount lustre-MDT0000 complete [ 791.283467] LustreError: 27256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786627890 with bad export cookie 6091054840282803551 [ 791.293634] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 791.532802] LustreError: 23706:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 791.549413] LustreError: 23706:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 791.851100] Lustre: server umount lustre-MDT0001 complete [ 798.218732] Lustre: server umount lustre-OST0000 complete [ 802.260657] Lustre: server umount lustre-OST0001 complete [ 819.122962] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 828.585233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 829.060469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 829.074900] Lustre: Skipped 3 previous similar messages [ 833.963704] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 842.446228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 848.136541] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 851.765930] Lustre: 30198:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 858.789703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 866.611271] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 867.621820] LustreError: 30553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 867.663504] LustreError: 30553:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 876.217272] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 884.039530] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 890.838281] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 895.084745] Lustre: 32078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 902.115970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 902.118549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 902.132328] Lustre: Skipped 5 previous similar messages [ 902.149103] Lustre: Skipped 9 previous similar messages [ 908.226895] Lustre: server umount lustre-MDT0000 complete [ 912.111821] LustreError: 30201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628011 with bad export cookie 6091054840282804839 [ 912.118843] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 912.121707] LustreError: 30201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 912.659284] Lustre: server umount lustre-MDT0001 complete [ 919.387620] Lustre: server umount lustre-OST0000 complete [ 923.781695] Lustre: server umount lustre-OST0001 complete [ 933.199404] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 09:33:50 (1786628030) [ 970.911250] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 981.444249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 988.584487] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 997.360810] LustreError: 34910:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 997.387947] LustreError: 34910:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 1000.689724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1009.278921] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1014.488953] Lustre: 36059:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1024.363988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1033.356215] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1043.111302] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1051.086725] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1058.992805] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1062.637883] Lustre: 37940:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1068.519576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1068.534357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1068.551373] Lustre: Skipped 3 previous similar messages [ 1068.567490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1068.578144] Lustre: Skipped 4 previous similar messages [ 1074.142834] Lustre: server umount lustre-MDT0000 complete [ 1078.159284] LustreError: 36443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628177 with bad export cookie 6091054840282805952 [ 1078.192288] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1078.193209] LustreError: 36443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1078.628264] Lustre: server umount lustre-MDT0001 complete [ 1092.995486] Lustre: server umount lustre-OST0000 complete [ 1095.137299] Lustre: 3654:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786628178/real 1786628178] req@ffff8a8e29cf4e00 x1873414428387456/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786628194 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1097.152738] Lustre: server umount lustre-OST0001 complete [ 1106.964583] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 09:36:44 (1786628204) [ 1124.338833] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 1135.559816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1136.136464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1136.140382] Lustre: Skipped 7 previous similar messages [ 1141.436921] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1150.578947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1156.773867] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1159.938582] Lustre: 41229:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1167.714648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1174.225245] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1183.685237] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1192.133910] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1201.619364] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1205.729688] Lustre: 43108:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1207.002971] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1207.005673] Lustre: Skipped 1 previous similar message [ 1211.550796] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.17@tcp (stopping) [ 1211.575262] Lustre: Skipped 4 previous similar messages [ 1212.898069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1212.914869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1212.927479] Lustre: Skipped 4 previous similar messages [ 1213.230489] Lustre: server umount lustre-MDT0000 complete [ 1217.488975] LustreError: 40061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628316 with bad export cookie 6091054840282807058 [ 1217.494363] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1217.505810] LustreError: 40061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1224.027276] Lustre: server umount lustre-MDT0001 complete [ 1232.002768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1242.596460] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x5487c61afe7aff5d [ 1242.616739] Lustre: MGC192.168.201.117@tcp: Connection restored to 0@lo (at 0@lo) [ 1247.918903] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1253.359254] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1253.368644] LustreError: 43982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1253.388659] LustreError: 43982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 39 previous similar messages [ 1256.685548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1262.088625] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1262.588357] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1262.599564] Lustre: Skipped 1 previous similar message [ 1269.982971] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1271.689297] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1277.358418] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1279.142690] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1287.649990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1290.026894] Lustre: server umount lustre-MDT0000 complete [ 1294.300491] LustreError: 40059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628393 with bad export cookie 6091054840282808157 [ 1294.307328] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1294.315439] LustreError: 40059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1294.812664] Lustre: server umount lustre-MDT0001 complete [ 1309.602042] Lustre: server umount lustre-OST0000 complete [ 1323.373150] Lustre: server umount lustre-OST0001 complete [ 1333.764941] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 09:40:31 (1786628431) [ 1350.384746] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 1366.088662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1373.449963] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1383.538559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1388.690301] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1391.826308] Lustre: 48872:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1400.141672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1407.610884] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1416.707592] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1425.298463] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1433.334121] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1437.948484] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1455.587173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1455.603922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1455.621025] Lustre: Skipped 9 previous similar messages [ 1468.384488] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1468.407076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1468.422075] Lustre: Skipped 22 previous similar messages [ 1468.652820] Lustre: server umount lustre-MDT0000 complete [ 1472.565569] LustreError: 47702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628571 with bad export cookie 6091054840282808976 [ 1472.566557] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1472.586081] LustreError: 47702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1473.204995] Lustre: server umount lustre-MDT0001 complete [ 1487.210621] Lustre: server umount lustre-OST0000 complete [ 1489.120134] Lustre: 3654:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786628572/real 1786628572] req@ffff8a8e29cff480 x1873414428538368/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786628588 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1492.139239] Lustre: server umount lustre-OST0001 complete [ 1503.951958] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 09:43:21 (1786628601) [ 1507.007792] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1508.857690] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 09:43:26 (1786628606) [ 1520.260136] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 1530.972365] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 1554.553754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1554.874884] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1554.986877] Lustre: lustre-MDT0000: new disk, initializing [ 1555.078945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1560.171734] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1570.349109] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1577.761434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1578.103712] Lustre: lustre-OST0000: new disk, initializing [ 1578.111357] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1578.117703] Lustre: Skipped 1 previous similar message [ 1578.139529] Lustre: 54949:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1579.596789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1579.601602] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1579.640921] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1585.252638] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1596.238775] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1598.552642] Lustre: DEBUG MARKER: create test files [ 1639.174672] Lustre: server umount lustre-OST0000 complete [ 1643.123742] Lustre: server umount lustre-MDT0000 complete [ 1654.805819] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 1658.151226] Key type lgssc unregistered [ 1658.459569] LNet: 56584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1658.464502] LNetError: 56584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1658.497153] LNet: Removed LNI 192.168.201.117@tcp [ 1659.483277] Key type .llcrypt unregistered [ 1659.491857] Key type ._llcrypt unregistered [ 1692.166483] Key type ._llcrypt registered [ 1692.172856] Key type .llcrypt registered [ 1692.299368] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 1694.203193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1694.234743] alg: No test for adler32 (adler32-zlib) [ 1695.297345] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 1695.542442] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 1697.192232] Key type lgssc registered [ 1698.735351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1707.892676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1707.928880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1709.479229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1714.194733] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1721.897125] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1729.792986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1730.092697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1734.205391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1734.367201] Lustre: 59319:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1734.391153] Lustre: 59319:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1734.406891] Lustre: 59319:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1734.418241] Lustre: 59319:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1734.429324] Lustre: 59319:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1734.440083] Lustre: 59319:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1737.477615] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1745.968707] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1751.485462] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1756.983861] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1764.833486] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1764.838672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1764.872199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1769.953700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1771.078310] Lustre: server umount lustre-OST0000 complete [ 1775.392616] Lustre: server umount lustre-MDT0000 complete [ 1787.170903] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 1790.797621] Key type lgssc unregistered [ 1791.140140] LNet: 60928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1791.146431] LNetError: 60928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1791.163044] LNet: Removed LNI 192.168.201.117@tcp [ 1792.316802] Key type .llcrypt unregistered [ 1792.318650] Key type ._llcrypt unregistered [ 1824.046161] Key type ._llcrypt registered [ 1824.052127] Key type .llcrypt registered [ 1824.206767] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 1825.613276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1825.721767] alg: No test for adler32 (adler32-zlib) [ 1826.914479] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 1827.368746] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 1829.160221] Key type lgssc registered [ 1830.867804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1840.052368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1840.081553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1841.777416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1846.801531] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1854.876933] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1861.844873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1862.319301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1866.434265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1866.533507] Lustre: 63761:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1866.543253] Lustre: 63761:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1866.555614] Lustre: 63761:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1866.564435] Lustre: 63761:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1866.571684] Lustre: 63761:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1866.581511] Lustre: 63761:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1869.743764] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1877.569600] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1883.220934] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1897.441818] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1897.471356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1898.761569] Lustre: server umount lustre-OST0000 complete [ 1902.672676] Lustre: server umount lustre-MDT0000 complete [ 1914.189188] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 1916.921442] Key type lgssc unregistered [ 1917.288788] LNet: 65322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1917.308362] LNetError: 65322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1917.322069] LNet: Removed LNI 192.168.201.117@tcp [ 1918.407648] Key type .llcrypt unregistered [ 1918.410705] Key type ._llcrypt unregistered [ 1936.471188] Key type ._llcrypt registered [ 1936.474558] Key type .llcrypt registered [ 1936.563334] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 1953.441548] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 1954.756591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1954.982921] alg: No test for adler32 (adler32-zlib) [ 1956.171552] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 1956.530526] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 1958.232180] Key type lgssc registered [ 1959.379832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1998.027657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1998.060192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1999.310389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1999.342207] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1999.407258] Lustre: lustre-MDT0000: new disk, initializing [ 1999.474559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1999.493902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2004.653480] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2015.462684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2015.576494] Lustre: 68476:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2015.632227] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2015.637515] Lustre: Skipped 1 previous similar message [ 2015.730384] Lustre: lustre-MDT0001: new disk, initializing [ 2015.845250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2015.905891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2015.919911] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2021.093898] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2032.995534] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2040.157344] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2048.200826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2048.445863] Lustre: lustre-OST0000: new disk, initializing [ 2048.452481] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2048.469071] Lustre: 70448:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2048.618297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2056.397386] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2058.329981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2058.344108] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2058.447561] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2066.828479] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2072.638307] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2072.915948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2078.279424] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2078.463587] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2084.323122] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2084.325864] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2084.340380] LustreError: Skipped 1 previous similar message [ 2084.346386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2084.365302] Lustre: Skipped 1 previous similar message [ 2084.391872] Lustre: Skipped 1 previous similar message [ 2086.648571] Lustre: server umount lustre-OST0000 complete [ 2092.512826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2092.522180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2092.539702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2094.575605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2094.586191] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2097.035367] Lustre: server umount lustre-MDT0000 complete [ 2101.157505] LustreError: 71988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629200 with bad export cookie 13408311909652730336 [ 2101.165360] LustreError: 71988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2101.171411] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2101.636924] Lustre: server umount lustre-MDT0001 complete [ 2110.739427] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 09:53:28 (1786629208) [ 2120.600324] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 2161.610578] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 2173.010562] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 2214.541799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2214.736889] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2214.761832] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2214.823195] Lustre: lustre-MDT0000: new disk, initializing [ 2214.886260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2214.905505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2220.603698] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2231.296182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2231.486921] Lustre: 76466:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2231.607388] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2231.613341] Lustre: Skipped 1 previous similar message [ 2231.740622] Lustre: lustre-MDT0001: new disk, initializing [ 2231.857495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2231.889446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2231.904882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2236.454123] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2246.319618] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2252.880303] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2259.897136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2260.336112] Lustre: lustre-OST0000: new disk, initializing [ 2260.346961] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2260.361974] Lustre: 78436:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2260.501913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2266.946028] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2268.339357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2268.364818] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2268.456668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2276.186800] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2280.998842] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2281.312297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2286.514934] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2286.728569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2289.134921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2289.137398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2289.145314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2289.145520] Lustre: Skipped 1 previous similar message [ 2289.150165] LustreError: Skipped 1 previous similar message [ 2289.162429] Lustre: Skipped 1 previous similar message [ 2294.251160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2294.261172] Lustre: Skipped 2 previous similar messages [ 2294.708772] Lustre: server umount lustre-OST0000 complete [ 2298.354260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2298.365134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2304.484749] LustreError: 78124:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2304.524354] LustreError: 78124:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2304.567340] Lustre: server umount lustre-MDT0000 complete [ 2308.982626] LustreError: 76459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629407 with bad export cookie 13408311909652731218 [ 2308.992526] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2308.994448] LustreError: 76459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2309.419583] Lustre: server umount lustre-MDT0001 complete [ 2318.600433] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 09:56:56 (1786629416) [ 2325.385259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2325.782993] LustreError: 80851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2325.865960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2330.747273] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2331.106434] LustreError: 80852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2335.203849] LustreError: 80851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2340.103936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2340.321898] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2340.331130] Lustre: Skipped 3 previous similar messages [ 2345.384928] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2353.177553] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2359.581802] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2367.561923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2368.067155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2368.079775] Lustre: Skipped 1 previous similar message [ 2374.542349] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2382.465152] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2386.482283] Lustre: 83524:0:(genops.c:1795:obd_export_evict_by_uuid()) MGS: evicting 06fe7762-e5cc-4886-9aee-1105ae07ce2e at adminstrative request [ 2390.517505] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2390.535376] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2400.741321] Lustre: MGS: Client 06fe7762-e5cc-4886-9aee-1105ae07ce2e (at 0@lo) reconnecting [ 2400.749559] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0xba13e4a43c7d98c4 to 0xba13e4a43c7d9b64 [ 2400.763674] Lustre: MGC192.168.201.117@tcp: Connection restored to 0@lo (at 0@lo) [ 2423.542210] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2423.720303] Lustre: lustre-OST0001: new disk, initializing [ 2423.724531] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2423.729678] Lustre: 83916:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2429.651891] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2430.054107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2430.066884] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2430.174780] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2441.833567] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2450.924378] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2450.927760] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2450.935330] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2450.961232] Lustre: Skipped 2 previous similar messages [ 2452.503632] Lustre: server umount lustre-OST0001 complete [ 2456.037762] LustreError: 82672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2456.077229] LustreError: 82672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2464.030269] Lustre: server umount lustre-OST0000 complete [ 2471.404673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2471.451695] Lustre: Skipped 3 previous similar messages [ 2475.700918] Lustre: server umount lustre-MDT0000 complete [ 2476.513687] LustreError: 80852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2476.535755] LustreError: 80852:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2480.761670] LustreError: 80832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629579 with bad export cookie 13408311909652732772 [ 2480.770761] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2480.775368] LustreError: 80832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2481.098296] Lustre: server umount lustre-MDT0001 complete [ 2491.176204] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 09:59:48 (1786629588) [ 2493.546586] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2495.880805] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 09:59:53 (1786629593) [ 2498.508594] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2500.686788] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 09:59:58 (1786629598) [ 2502.384926] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2504.444087] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 10:00:02 (1786629602) [ 2506.105860] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2507.770626] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 10:00:05 (1786629605) [ 2509.639423] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2517.180678] loop: module loaded [ 2517.213204] loop0: detected capacity change from 0 to 4294967296 [ 2517.294042] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2519.186081] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2522.987559] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2528.650380] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2558.804613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2558.866346] LustreError: 88039:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2558.870454] LustreError: 88039:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2558.875653] LustreError: 88039:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2558.880555] LustreError: 88039:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2579.414160] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 2591.393384] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 2637.458171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2637.847214] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2637.865947] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2637.986294] Lustre: lustre-MDT0000: new disk, initializing [ 2638.179869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2638.183274] Lustre: Skipped 1 previous similar message [ 2638.205453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2643.562465] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2657.065442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2657.189115] Lustre: 90919:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2657.467606] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2662.991496] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2673.630662] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2681.601683] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2691.269264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2691.704671] Lustre: 92888:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2698.470298] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2701.947680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2702.130490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2708.014556] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2712.990076] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2713.188421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2718.604177] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2718.946369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2722.787349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2722.799547] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2722.801139] LustreError: Skipped 1 previous similar message [ 2722.811186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2722.811322] Lustre: Skipped 1 previous similar message [ 2722.820545] Lustre: Skipped 5 previous similar messages [ 2736.096111] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2736.245737] Lustre: server umount lustre-OST0000 complete [ 2746.297405] Lustre: server umount lustre-MDT0000 complete [ 2748.386718] LustreError: 90926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2748.431341] LustreError: 90926:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2751.188626] LustreError: 90912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629850 with bad export cookie 13408311909652733388 [ 2751.198671] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2751.209171] LustreError: 90912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2751.867634] Lustre: server umount lustre-MDT0001 complete [ 2763.075666] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 10:04:19 (1786629859) [ 2772.400396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2772.923951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-60 [ 2772.934115] Lustre: Skipped 2 previous similar messages [ 2777.664911] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2782.188906] LustreError: 95305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2782.204202] LustreError: 95305:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2786.328094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2792.759809] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2801.765932] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2808.555034] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2815.848637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2823.539931] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2831.116453] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2837.111728] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2837.443178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2842.227511] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2842.519711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2849.970746] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2850.238800] Lustre: lustre-OST0001: new disk, initializing [ 2850.242378] Lustre: Skipped 2 previous similar messages [ 2850.254643] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2850.264255] Lustre: Skipped 3 previous similar messages [ 2850.272374] Lustre: 98676:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2858.532448] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2860.154046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2860.165536] Lustre: Skipped 2 previous similar messages [ 2860.176855] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2860.356063] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2870.266223] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2876.609252] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2876.899985] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2882.391690] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2882.652735] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2904.200416] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2930.514489] Lustre: Failing over lustre-MDT0000 [ 2930.967147] Lustre: server umount lustre-MDT0000 complete [ 2932.205819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2932.209423] LustreError: 97189:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2932.227560] Lustre: Skipped 4 previous similar messages [ 2932.266457] LustreError: 97189:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2948.576312] Lustre: 66700:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786630031/real 1786630031] req@00000000d1dc7c33 x1873416348258432/t0(0) o400->MGC192.168.201.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786630047 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2948.599834] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2957.432879] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2957.436144] LDISKFS-fs (dm-0): recovery complete [ 2957.442915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2958.624155] LustreError: 101495:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2958.637347] LustreError: 101495:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000392b6182 x1873416348264192/t0(0) o250->MGC192.168.201.117@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1786630057 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2958.664944] LustreError: 101495:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 2958.827903] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xba13e4a43c7e65df [ 2958.844762] Lustre: MGC192.168.201.117@tcp: Connection restored to 0@lo (at 0@lo) [ 2959.194443] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2963.083095] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2963.084756] Lustre: 99933:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2964.284774] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2964.456868] Lustre: 95304:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2964.470023] Lustre: 95304:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 1 previous similar message [ 2964.507448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2964.591149] LustreError: 101527:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2964.922317] LustreError: 101527:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2965.242939] LustreError: 101527:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2965.259758] LustreError: 101527:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 1 previous similar message [ 2965.578213] LustreError: 101527:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2965.581848] LustreError: 101527:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 1 previous similar message [ 2965.619014] Lustre: 101527:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2965.642035] Lustre: 101527:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 7 previous similar messages [ 2966.305119] LustreError: 101527:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2966.313104] LustreError: 101527:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 2 previous similar messages [ 2966.643332] LustreError: 101527:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2966.653665] LustreError: 101527:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 2 previous similar messages [ 2967.632958] Lustre: 101527:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2967.667088] Lustre: 101527:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 5 previous similar messages [ 2968.337096] LustreError: 101527:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2968.346741] LustreError: 101527:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 5 previous similar messages [ 2968.672300] LustreError: 101527:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2968.685536] LustreError: 101527:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 5 previous similar messages [ 2971.899914] Lustre: 101527:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2971.911500] Lustre: 101527:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 12 previous similar messages [ 2972.558691] LustreError: 101527:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2972.581455] LustreError: 101527:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 12 previous similar messages [ 2972.912140] LustreError: 101527:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2972.924329] LustreError: 101527:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 12 previous similar messages [ 2979.937791] Lustre: 101527:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2979.958505] Lustre: 101527:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 24 previous similar messages [ 2980.601160] LustreError: 101527:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2980.612707] LustreError: 101527:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 24 previous similar messages [ 2980.937164] LustreError: 101527:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2980.946030] LustreError: 101527:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 24 previous similar messages [ 2995.937917] Lustre: 101527:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2995.955599] Lustre: 101527:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 49 previous similar messages [ 2996.611509] LustreError: 101527:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2996.636166] LustreError: 101527:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 49 previous similar messages [ 2996.960524] LustreError: 101527:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2996.974756] LustreError: 101527:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 49 previous similar messages [ 3023.184697] Lustre: 101527:0:(ldlm_lib.c:2394:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 3023.209205] Lustre: 101527:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3023.233640] Lustre: 101527:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 27133e7a-42a3-442f-aa15-359e0ffad604@192.168.201.17@tcp [ 3023.272817] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3023.289128] Lustre: 101527:0:(ldlm_lib.c:1914:abort_req_replay_queue()) @@@ aborted: req@00000000e27a4179 x1873416330222208/t0(8589934779) o101->27133e7a-42a3-442f-aa15-359e0ffad604@192.168.201.17@tcp:397/0 lens 592/0 e 0 to 0 dl 1786630132 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 3023.307186] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3023.324811] Lustre: lustre-MDT0000: Denying connection for new client 27133e7a-42a3-442f-aa15-359e0ffad604 (at 192.168.201.17@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 3023.345499] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3023.389342] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3023.413406] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 3023.428135] Lustre: Skipped 2 previous similar messages [ 3023.465604] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:48 to 0x2c0000401:545) [ 3023.468899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:79 to 0x280000401:577) [ 3042.793069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3042.807384] LustreError: Skipped 1 previous similar message [ 3042.818442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3042.827206] Lustre: Skipped 7 previous similar messages [ 3047.640647] Lustre: server umount lustre-OST0000 complete [ 3058.679791] Lustre: server umount lustre-OST0001 complete [ 3064.290994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3064.314040] Lustre: Skipped 5 previous similar messages [ 3069.544648] Lustre: server umount lustre-MDT0000 complete [ 3071.972654] LustreError: 95299:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3071.989906] LustreError: 95299:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 41 previous similar messages [ 3073.665917] LustreError: 95285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630172 with bad export cookie 13408311909652784607 [ 3073.668304] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3073.674631] LustreError: 95285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3074.328261] Lustre: server umount lustre-MDT0001 complete [ 3083.835534] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 10:09:41 (1786630181) [ 3091.667419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3091.679438] Lustre: *** cfs_fail_loc=197, val=0*** [ 3107.298299] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3112.416607] LustreError: 103508:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.117@tcp: failed processing log, type 4: rc = -110 [ 3117.626918] Lustre: *** cfs_fail_loc=197, val=0*** [ 3117.633264] Lustre: Skipped 65 previous similar messages [ 3138.208529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3138.221651] Lustre: Skipped 4 previous similar messages [ 3146.414402] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3155.361667] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3159.040504] Lustre: server umount lustre-OST0000 complete [ 3169.215300] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 10:11:06 (1786630266) [ 3195.163158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3202.480563] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3213.419361] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3217.712359] LustreError: 105159:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3217.723203] LustreError: 105159:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3217.733413] LustreError: 105159:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 3218.118975] Lustre: server umount lustre-MDT0000 complete [ 3258.125786] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 3269.492169] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 3309.700451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3310.088485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3310.138302] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3310.278716] Lustre: lustre-MDT0000: new disk, initializing [ 3310.454475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3316.093199] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3328.334858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3328.533960] Lustre: 109337:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3329.010271] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3335.445610] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3349.081896] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3356.768203] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3365.377949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3365.632128] Lustre: 111305:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3373.106727] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3375.802408] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3383.342571] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3389.814817] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3390.139674] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3395.385229] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3395.672465] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3401.711323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3401.711428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3401.712551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3401.712557] Lustre: Skipped 10 previous similar messages [ 3401.750220] Lustre: Skipped 2 previous similar messages [ 3401.794558] LustreError: Skipped 5 previous similar messages [ 3404.307223] Lustre: server umount lustre-OST0000 complete [ 3423.200415] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3423.714927] Lustre: server umount lustre-MDT0000 complete [ 3426.273152] LustreError: 109347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3426.283743] LustreError: 109347:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3428.009435] LustreError: 110131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630527 with bad export cookie 13408311909652795289 [ 3428.024048] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3428.027153] LustreError: 110131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3428.470093] Lustre: server umount lustre-MDT0001 complete [ 3437.324038] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 10:15:35 (1786630535) [ 3448.014314] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 3453.143849] Key type lgssc unregistered [ 3453.405814] LNet: 113798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3453.436233] LNetError: 113798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3453.468903] LNet: Removed LNI 192.168.201.117@tcp [ 3454.272895] Key type .llcrypt unregistered [ 3454.276696] Key type ._llcrypt unregistered [ 3475.661894] Key type ._llcrypt registered [ 3475.669113] Key type .llcrypt registered [ 3475.859567] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 3494.354623] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 3496.324096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3496.448063] alg: No test for adler32 (adler32-zlib) [ 3497.585194] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 3497.786272] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 3499.440179] Key type lgssc registered [ 3500.447730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3544.522296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3544.554205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3545.932245] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3545.989815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3546.129621] Lustre: lustre-MDT0000: new disk, initializing [ 3546.316620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3546.365971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3552.828137] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3565.462433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3565.848222] Lustre: lustre-OST0000: new disk, initializing [ 3565.852639] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3565.861241] Lustre: Skipped 1 previous similar message [ 3565.870675] Lustre: 117977:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3565.963535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3572.310792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3572.317115] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3572.356831] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3573.505629] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3585.789686] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3586.040434] Lustre: lustre-OST0001: new disk, initializing [ 3586.046586] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3586.052069] Lustre: 118842:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3586.132488] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3593.569881] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3594.957291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3594.971501] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3595.074721] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3605.519641] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3605.564255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3605.578454] Lustre: Skipped 1 previous similar message [ 3610.594161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3610.625939] Lustre: Skipped 1 previous similar message [ 3611.878909] Lustre: server umount lustre-MDT0000 complete [ 3620.929913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3627.392275] LustreError: 117976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630726 with bad export cookie 4733507976107550670 [ 3627.393800] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3627.402263] LustreError: 117976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3627.535977] Lustre: server umount lustre-OST0000 complete [ 3633.055266] Lustre: server umount lustre-OST0001 complete [ 3642.789963] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 10:19:00 (1786630740) [ 3670.666984] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 10:19:28 (1786630768) [ 3682.774731] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 3711.118994] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 10:20:08 (1786630808) [ 3756.211572] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 3766.657228] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 3810.441374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3810.671716] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3810.698628] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3810.764681] Lustre: lustre-MDT0000: new disk, initializing [ 3810.817387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3810.834337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3815.224488] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3825.859451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3825.949843] Lustre: 126595:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3826.017436] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3826.021544] Lustre: Skipped 1 previous similar message [ 3826.114338] Lustre: lustre-MDT0001: new disk, initializing [ 3826.217388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3826.271082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3826.284125] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3830.833770] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3840.641866] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3846.448387] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3853.680179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3854.057832] Lustre: lustre-OST0000: new disk, initializing [ 3854.063740] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3854.068602] Lustre: 128569:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3854.159200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3861.065881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3861.083801] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3861.193555] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3861.854315] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3871.845300] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3876.767980] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3877.041786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3882.153458] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3882.374659] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3887.073344] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3887.082165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3887.091335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3887.105808] Lustre: Skipped 2 previous similar messages [ 3887.124302] Lustre: Skipped 1 previous similar message [ 3890.611737] Lustre: server umount lustre-OST0000 complete [ 3897.339296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3897.341980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3897.351209] Lustre: Skipped 1 previous similar message [ 3897.370739] Lustre: Skipped 1 previous similar message [ 3901.622642] Lustre: server umount lustre-MDT0000 complete [ 3906.438582] LustreError: 127390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786631005 with bad export cookie 4733507976107551895 [ 3906.463175] LustreError: 127390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3906.469098] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3906.941400] Lustre: server umount lustre-MDT0001 complete [ 3916.886328] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 10:23:34 (1786631014) [ 3924.905513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3925.602620] LustreError: 130981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3925.622663] LustreError: 130981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3925.686774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3931.107193] LustreError: 130982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3931.738604] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3935.213605] LustreError: 130981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3940.322422] LustreError: 130982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3941.105067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3946.786511] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3955.524723] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3962.522522] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3970.595097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3971.127234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3971.136355] Lustre: Skipped 1 previous similar message [ 3978.650834] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3987.984538] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4003.508470] Lustre: *** cfs_fail_loc=159, val=0*** [ 4003.520191] Lustre: Skipped 4 previous similar messages [ 4018.874303] Lustre: lustre-MDT0000: Client 187fc13c-d024-4970-8a3c-c59d5c4d563c (at 192.168.201.17@tcp) reconnecting [ 4021.026406] Lustre: *** cfs_fail_loc=159, val=0*** [ 4021.040722] Lustre: Skipped 2 previous similar messages [ 4037.359928] Lustre: lustre-MDT0000: Client 187fc13c-d024-4970-8a3c-c59d5c4d563c (at 192.168.201.17@tcp) reconnecting [ 4046.821894] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4046.831204] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4046.840630] LustreError: Skipped 1 previous similar message [ 4046.884582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4046.900312] Lustre: Skipped 1 previous similar message [ 4051.577406] Lustre: server umount lustre-OST0000 complete [ 4057.063046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4057.076166] Lustre: Skipped 1 previous similar message [ 4057.086746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4057.096797] Lustre: Skipped 1 previous similar message [ 4069.344151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4069.753633] Lustre: server umount lustre-MDT0000 complete [ 4071.414616] LustreError: 130981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4071.442937] LustreError: 130981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4073.366592] LustreError: 130962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786631172 with bad export cookie 4733507976107552770 [ 4073.373485] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4073.383148] LustreError: 130962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4073.768539] Lustre: server umount lustre-MDT0001 complete [ 4086.344345] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 4090.588434] Key type lgssc unregistered [ 4091.001391] LNet: 134788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4091.010984] LNetError: 134788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4091.041351] LNet: Removed LNI 192.168.201.117@tcp [ 4091.898172] Key type .llcrypt unregistered [ 4091.900284] Key type ._llcrypt unregistered [ 4124.420449] Key type ._llcrypt registered [ 4124.424387] Key type .llcrypt registered [ 4124.583896] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 4125.876927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4126.200080] alg: No test for adler32 (adler32-zlib) [ 4127.443938] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 4127.746692] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 4129.450340] Key type lgssc registered [ 4130.902503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4139.169795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4139.204577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4140.707185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4145.425884] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4150.249247] LustreError: 136217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4150.270402] LustreError: 136217:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4154.310720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4154.565830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4157.865321] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4167.021349] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4174.163956] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4181.470441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4182.013160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4185.085208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4188.318372] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4198.613585] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4211.485326] Lustre: *** cfs_fail_loc=159, val=0*** [ 4226.763512] Lustre: lustre-MDT0000: Client c3b13e88-3a18-4b0f-959d-498bc6f5a37f (at 192.168.201.17@tcp) reconnecting [ 4228.858056] Lustre: *** cfs_fail_loc=159, val=0*** [ 4245.211260] Lustre: lustre-MDT0001: Client c3b13e88-3a18-4b0f-959d-498bc6f5a37f (at 192.168.201.17@tcp) reconnecting [ 4246.595155] Lustre: *** cfs_fail_loc=159, val=0*** [ 4246.598372] Lustre: Skipped 3 previous similar messages [ 4262.084313] Lustre: lustre-MDT0001: Client c3b13e88-3a18-4b0f-959d-498bc6f5a37f (at 192.168.201.17@tcp) reconnecting [ 4268.551299] Lustre: 139262:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4271.497572] Lustre: *** cfs_fail_loc=159, val=0*** [ 4271.500282] Lustre: Skipped 4 previous similar messages [ 4287.150530] Lustre: lustre-MDT0000: Client 239f1b23-d5eb-4c42-96ca-22bf9710415e (at 192.168.201.17@tcp) reconnecting [ 4289.398220] Lustre: *** cfs_fail_loc=159, val=0*** [ 4289.413024] Lustre: Skipped 17 previous similar messages [ 4305.082833] Lustre: lustre-MDT0000: Client 239f1b23-d5eb-4c42-96ca-22bf9710415e (at 192.168.201.17@tcp) reconnecting [ 4306.928390] Lustre: 139553:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4327.425517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4327.439107] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4327.463557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4331.924658] Lustre: server umount lustre-OST0000 complete [ 4337.637869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4337.659957] Lustre: Skipped 1 previous similar message [ 4337.674095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4337.687121] Lustre: Skipped 1 previous similar message [ 4341.738718] Lustre: server umount lustre-MDT0000 complete [ 4342.760400] LustreError: 139323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4342.791685] LustreError: 139323:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4346.434963] LustreError: 136198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786631445 with bad export cookie 17275957947568586175 [ 4346.462721] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4346.463596] LustreError: 136198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4346.845751] Lustre: server umount lustre-MDT0001 complete [ 4358.942702] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 4362.172370] Key type lgssc unregistered [ 4362.581797] LNet: 140634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4362.587901] LNetError: 140634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4362.620060] LNet: Removed LNI 192.168.201.117@tcp [ 4363.503398] Key type .llcrypt unregistered [ 4363.505728] Key type ._llcrypt unregistered [ 4391.300564] Key type ._llcrypt registered [ 4391.302991] Key type .llcrypt registered [ 4391.438786] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 4392.664450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4392.834343] alg: No test for adler32 (adler32-zlib) [ 4394.080449] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 4394.460283] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 4396.168634] Key type lgssc registered [ 4397.491257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4406.198035] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4406.212223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4407.619483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4413.366800] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4418.021970] LustreError: 142050:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4418.061258] LustreError: 142050:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4422.574837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4422.820821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4428.316870] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4439.126373] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4447.429920] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4454.730485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4455.028869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4461.348462] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4467.325218] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4470.193052] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4478.599136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4487.266146] Lustre: 144655:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4490.903749] Lustre: 144703:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4492.457254] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4493.445670] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4497.327850] Lustre: 144849:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4498.396923] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4499.221822] Lustre: 144947:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4504.553669] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4504.563501] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4504.583660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4504.603754] Lustre: Skipped 1 previous similar message [ 4509.668584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4509.681592] Lustre: Skipped 1 previous similar message [ 4514.804255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4514.813833] Lustre: Skipped 1 previous similar message [ 4518.370764] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4518.689888] Lustre: server umount lustre-OST0000 complete [ 4525.025435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4525.051392] Lustre: Skipped 1 previous similar message [ 4525.064692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4525.076041] Lustre: Skipped 1 previous similar message [ 4530.154360] LustreError: 142051:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4530.193344] LustreError: 142051:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4530.813189] Lustre: server umount lustre-MDT0000 complete [ 4535.273335] LustreError: 142793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4535.311174] LustreError: 142793:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4536.323259] LustreError: 142787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786631635 with bad export cookie 361648185002246303 [ 4536.324249] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4536.348495] LustreError: 142787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4536.985837] Lustre: server umount lustre-MDT0001 complete [ 4550.266505] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 4553.519714] Key type lgssc unregistered [ 4553.843966] LNet: 146013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4553.852547] LNetError: 146013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4553.877426] LNet: Removed LNI 192.168.201.117@tcp [ 4554.986353] Key type .llcrypt unregistered [ 4554.988368] Key type ._llcrypt unregistered [ 4582.042885] Key type ._llcrypt registered [ 4582.054515] Key type .llcrypt registered [ 4582.210812] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 4583.743450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4583.800542] alg: No test for adler32 (adler32-zlib) [ 4585.083285] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 4585.480750] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 4587.233499] Key type lgssc registered [ 4588.833112] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4599.310995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4599.323518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4600.916925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4605.452248] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4610.035126] LustreError: 147430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4610.059110] LustreError: 147430:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4613.466529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4613.936365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4618.666642] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4626.278586] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4632.804616] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4640.323498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4640.809054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4645.929314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4648.459854] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4656.981830] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4658.241502] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4663.597550] Lustre: *** cfs_fail_loc=159, val=0*** [ 4663.603500] Lustre: Skipped 2 previous similar messages [ 4678.863914] Lustre: lustre-MDT0000: Client 36ad6c15-443a-4e03-9417-e9420de7be32 (at 192.168.201.17@tcp) reconnecting [ 4686.817835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4686.842752] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4686.873632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4688.872670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4688.878320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4688.900104] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4691.489362] Lustre: server umount lustre-OST0000 complete [ 4696.038048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4696.044605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4696.054693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4696.058596] Lustre: Skipped 1 previous similar message [ 4699.112579] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4699.116558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4699.134022] Lustre: Skipped 1 previous similar message [ 4704.235429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4704.244462] Lustre: Skipped 1 previous similar message [ 4710.368929] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4710.855842] Lustre: server umount lustre-MDT0000 complete [ 4713.442710] LustreError: 147429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4713.467906] LustreError: 147429:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4716.822440] LustreError: 150139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786631815 with bad export cookie 12842054919139729153 [ 4716.824749] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4716.837733] LustreError: 150139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4717.348206] Lustre: server umount lustre-MDT0001 complete [ 4732.176638] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 4737.179297] Key type lgssc unregistered [ 4737.552624] LNet: 151106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4737.561457] LNetError: 151106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4737.585492] LNet: Removed LNI 192.168.201.117@tcp [ 4738.872186] Key type .llcrypt unregistered [ 4738.876186] Key type ._llcrypt unregistered [ 4768.548503] Key type ._llcrypt registered [ 4768.554720] Key type .llcrypt registered [ 4768.738966] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 4770.443906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4770.522699] alg: No test for adler32 (adler32-zlib) [ 4772.181823] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 4772.585587] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 4774.328230] Key type lgssc registered [ 4775.613725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4783.713365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4783.730702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4785.277191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4790.798689] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4795.875857] LustreError: 152521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4795.892842] LustreError: 152521:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4800.949664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4800.995725] LustreError: 152522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4801.658981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4807.284499] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4816.248559] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4822.794824] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4831.348495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4831.767830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4839.755669] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4846.174978] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4848.621626] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4855.447909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4875.165862] Lustre: lustre-OST0000: evicting 266d0733-96a5-47d0-876c-79fefd2ac868 (at 192.168.201.17@tcp) by administrative request [ 4875.179505] Lustre: Skipped 1 previous similar message [ 4906.979095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4906.989957] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4906.994316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4907.034909] Lustre: Skipped 1 previous similar message [ 4912.099426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4912.117035] Lustre: Skipped 2 previous similar messages [ 4917.219142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4917.227022] Lustre: Skipped 1 previous similar message [ 4919.264130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4919.632476] Lustre: server umount lustre-OST0000 complete [ 4924.388358] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4924.404355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4924.452613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4927.467392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4929.979849] Lustre: server umount lustre-MDT0000 complete [ 4932.579905] LustreError: 153506:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4932.606493] LustreError: 153506:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4934.410584] LustreError: 152503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786632033 with bad export cookie 4067264226492715515 [ 4934.416264] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4934.425694] LustreError: 152503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4934.859810] Lustre: server umount lustre-MDT0001 complete [ 4948.271533] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 4951.887320] Key type lgssc unregistered [ 4952.248236] LNet: 156603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4952.259630] LNetError: 156603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4952.314539] LNet: Removed LNI 192.168.201.117@tcp [ 4953.320645] Key type .llcrypt unregistered [ 4953.322885] Key type ._llcrypt unregistered [ 5042.395163] Key type ._llcrypt registered [ 5042.400921] Key type .llcrypt registered [ 5042.556131] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 5043.736150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5043.887501] alg: No test for adler32 (adler32-zlib) [ 5045.194686] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 5045.492529] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 5047.184697] Key type lgssc registered [ 5048.469939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5057.277439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5057.331593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5058.928201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5064.172851] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5068.258074] LustreError: 159096:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5068.279728] LustreError: 159096:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5073.377489] LustreError: 159097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5074.384202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5075.141480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5081.008050] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5090.551754] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5096.674576] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5103.322561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5103.767295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5110.279841] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5118.338753] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5119.175195] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 5128.435958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 5138.913809] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5138.926054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5138.943922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5138.953541] Lustre: Skipped 1 previous similar message [ 5138.981810] Lustre: Skipped 1 previous similar message [ 5144.060464] Lustre: server umount lustre-OST0000 complete [ 5151.725687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5151.733586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5151.751325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5154.274352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5154.274763] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5155.931594] Lustre: server umount lustre-MDT0000 complete [ 5159.393214] LustreError: 159096:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5159.416240] LustreError: 159096:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5161.126382] LustreError: 159079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786632260 with bad export cookie 8952688714631357537 [ 5161.135261] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5161.160264] LustreError: 159079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5161.867215] Lustre: server umount lustre-MDT0001 complete [ 5175.346727] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5179.745488] Key type lgssc unregistered [ 5180.146890] LNet: 162664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5180.182276] LNetError: 162664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5181.230121] LNet: Removed LNI 192.168.201.117@tcp [ 5182.318125] Key type .llcrypt unregistered [ 5182.319929] Key type ._llcrypt unregistered [ 5209.978903] Key type ._llcrypt registered [ 5209.982925] Key type .llcrypt registered [ 5210.101487] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 5211.535275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5211.593452] alg: No test for adler32 (adler32-zlib) [ 5212.762880] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 5213.082923] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 5214.824339] Key type lgssc registered [ 5216.430396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5233.576796] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 10:45:31 (1786632331) [ 5253.118559] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 5265.851838] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 5307.034869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5307.115158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5308.500176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5308.530247] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5308.605580] Lustre: lustre-MDT0000: new disk, initializing [ 5308.692823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5308.735232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5313.963600] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5325.135262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5325.309648] Lustre: 166928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5325.446885] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5325.455256] Lustre: Skipped 1 previous similar message [ 5325.595929] Lustre: lustre-MDT0001: new disk, initializing [ 5325.809497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5325.873247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5325.895254] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5331.647675] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5342.008108] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5348.719658] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5356.382871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5356.730797] Lustre: lustre-OST0000: new disk, initializing [ 5356.733780] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5356.741294] Lustre: 168898:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5356.820908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5363.301848] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5365.858280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5365.872136] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5365.986874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5372.670464] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5386.727327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5386.746059] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5386.747035] Lustre: Skipped 1 previous similar message [ 5386.747951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5390.635635] Lustre: server umount lustre-OST0000 complete [ 5396.985599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5397.016899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5397.028155] Lustre: Skipped 2 previous similar messages [ 5402.083561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5402.097287] Lustre: Skipped 1 previous similar message [ 5406.178404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5406.189342] Lustre: Skipped 1 previous similar message [ 5409.248172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5409.615157] Lustre: server umount lustre-MDT0000 complete [ 5414.145486] LustreError: 166920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786632513 with bad export cookie 100457440573193886 [ 5414.148717] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5414.162810] LustreError: 166920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5414.680019] Lustre: server umount lustre-MDT0001 complete [ 5428.809775] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5432.528800] Key type lgssc unregistered [ 5432.978601] LNet: 170808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5432.990125] LNetError: 170808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5433.017408] LNet: Removed LNI 192.168.201.117@tcp [ 5433.891876] Key type .llcrypt unregistered [ 5433.894821] Key type ._llcrypt unregistered [ 5462.276298] Key type ._llcrypt registered [ 5462.280075] Key type .llcrypt registered [ 5462.431858] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 5463.880203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5463.910607] alg: No test for adler32 (adler32-zlib) [ 5465.073243] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 5465.316547] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 5466.986397] Key type lgssc registered [ 5468.620799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5479.410631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5479.447744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5481.231458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5487.286719] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5491.693083] LustreError: 172225:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5491.717359] LustreError: 172225:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5496.804882] LustreError: 172226:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5499.462280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5499.879856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5505.420321] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5515.049787] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5522.065366] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5530.956463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5531.571305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5539.896804] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5548.805913] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5551.045296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5559.825409] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5559.851332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5559.866949] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5559.881378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.902469] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5561.133710] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5561.175052] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5561.205055] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5561.222883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.241049] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5562.818775] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5562.824472] LustreError: 172245:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5562.835056] LustreError: 172245:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5562.841187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5562.850541] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5562.855528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.865446] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5563.965287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.248870] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5565.258760] Lustre: Skipped 1 previous similar message [ 5565.285384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5565.299438] Lustre: Skipped 1 previous similar message [ 5565.306577] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5565.313607] Lustre: Skipped 1 previous similar message [ 5565.321438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.336506] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5565.349220] Lustre: Skipped 1 previous similar message [ 5566.831739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.267873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.273675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.582498] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5570.597775] Lustre: Skipped 3 previous similar messages [ 5570.628768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5570.638935] Lustre: Skipped 3 previous similar messages [ 5570.648974] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5570.658170] Lustre: Skipped 3 previous similar messages [ 5570.664209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.679279] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5570.691168] Lustre: Skipped 3 previous similar messages [ 5571.944336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.323282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.338464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.652579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.888802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.869757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.025037] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5579.030629] Lustre: Skipped 6 previous similar messages [ 5579.052898] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5579.061171] Lustre: Skipped 6 previous similar messages [ 5579.064807] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5579.073510] Lustre: Skipped 6 previous similar messages [ 5579.077428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.090988] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5579.109923] Lustre: Skipped 6 previous similar messages [ 5580.166968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.141905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.439442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.552738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.906874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.028786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.134704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.292819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.444143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.605789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.790901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.304504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.685323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.978567] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5595.989114] Lustre: Skipped 13 previous similar messages [ 5596.010484] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5596.020417] Lustre: Skipped 13 previous similar messages [ 5596.029244] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5596.032331] Lustre: Skipped 13 previous similar messages [ 5596.047487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.065603] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5596.089764] Lustre: Skipped 13 previous similar messages [ 5597.194124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.421692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.531488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.555893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.837431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.927168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.104979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.285545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.545227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.733895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.971370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.145985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.501943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.826366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.033105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.088665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.261721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.423898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.704188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.902326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.165252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.311759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.658451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.866690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.986347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.069618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.134985] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5628.144349] Lustre: Skipped 26 previous similar messages [ 5628.175803] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5628.185710] Lustre: Skipped 26 previous similar messages [ 5628.190057] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5628.196341] Lustre: Skipped 26 previous similar messages [ 5628.205567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.224497] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5628.232470] Lustre: Skipped 26 previous similar messages [ 5629.532928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.490237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.691152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.297075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.488282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.752707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.007465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.257919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.712859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.838895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.746976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.766602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.936811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.851479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.074042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.972301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.145486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.129905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.259327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.343383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.515922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.420871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.414942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.739610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.889541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.054257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.205657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.444519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.504757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.652956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.068932] LustreError: 172245:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5664.081465] LustreError: 172245:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5664.090262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.171950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.122975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.210645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.703797] LustreError: 172245:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5668.717251] LustreError: 172245:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5668.732164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.236029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.550445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.439115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.495775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.813595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.691181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.972474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.114160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.479293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.078934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.446472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.403447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.822988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.867389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.085216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.121478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.079576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.057128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.312973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.314418] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5692.319654] Lustre: Skipped 54 previous similar messages [ 5692.332806] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5692.348250] Lustre: Skipped 54 previous similar messages [ 5692.359309] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5692.367825] Lustre: Skipped 54 previous similar messages [ 5692.372785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.385799] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5692.394833] Lustre: Skipped 54 previous similar messages [ 5693.474778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.370771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.239485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.528174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.774635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.079510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.461928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.792752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.426890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.781437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.182122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.235994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.467620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.888057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.986143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.615539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.173337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.711316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.144327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.453133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.863278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.987579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.131224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.119641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.542398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.758877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.645860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.537640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.728240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.138846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.312048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.855543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.264918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.833492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.735586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.713621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.784257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.953391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.178204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.461373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.641149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.118936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.277857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.197667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.345534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.436227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.610874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.015643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.543669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.977583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.123119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.290120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.504568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.537557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.683892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.681314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.582451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.580463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.663619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.771833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.874746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.797720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.897869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.990184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.136379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.127144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.258042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.199571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.345361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.219530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.157165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.161900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.134498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.258700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.196565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.154906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.171936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.355825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.694497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.659954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.069454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.348639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.569945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.785263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.963042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.374714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.557537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.857631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.164772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.326166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.358556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.406914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.644533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.640284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.004605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.081869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.060380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.253345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.171702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.160488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.078923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.013978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.994341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.956389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.906706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.890567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.928962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.184288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.298914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.246836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.372923] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5821.380314] Lustre: Skipped 110 previous similar messages [ 5821.396823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5821.407362] Lustre: Skipped 110 previous similar messages [ 5821.413035] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5821.417178] Lustre: Skipped 110 previous similar messages [ 5821.420421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.428988] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5821.431142] Lustre: Skipped 110 previous similar messages [ 5822.394853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.483491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.568704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.837838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.879338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.938424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.051741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.266437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.215122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.429682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.535885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.797583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.977467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.441113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.622507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.139542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.413686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.744658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.969307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.961333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.060632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.286274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.383193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.527884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.563982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.679202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.847471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.803921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.596635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.690964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.584347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.907487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.992358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.186278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.230750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.338330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.441399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.489730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.605854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.697962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.674317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.895962] LustreError: 172245:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5868.908122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.916595] LustreError: 172245:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5869.898659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.034616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.065242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.157564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.173914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.113684] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5915.129284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5918.693032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5918.716718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5918.725755] Lustre: Skipped 1 previous similar message [ 5920.856393] Lustre: server umount lustre-OST0000 complete [ 5928.939888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5928.951015] Lustre: Skipped 1 previous similar message [ 5934.050679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5934.058960] Lustre: Skipped 1 previous similar message [ 5938.146961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5938.155730] Lustre: Skipped 1 previous similar message [ 5940.192300] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5940.489434] Lustre: server umount lustre-MDT0000 complete [ 5943.269483] LustreError: 183425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5943.292084] LustreError: 183425:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5945.329985] LustreError: 172207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786633044 with bad export cookie 9881587024533970888 [ 5945.340222] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5945.354718] LustreError: 172207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5945.828246] Lustre: server umount lustre-MDT0001 complete [ 5960.029477] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5963.681598] Key type lgssc unregistered [ 5964.266492] LNet: 189964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5964.278267] LNetError: 189964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5964.326617] LNet: Removed LNI 192.168.201.117@tcp [ 5965.741701] Key type .llcrypt unregistered [ 5965.743534] Key type ._llcrypt unregistered [ 6000.520755] Key type ._llcrypt registered [ 6000.525330] Key type .llcrypt registered [ 6000.706149] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 6002.434363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6002.511782] alg: No test for adler32 (adler32-zlib) [ 6003.777937] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 6004.130988] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 6005.940129] Key type lgssc registered [ 6007.753517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6019.790807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6019.818916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6021.573774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6027.466384] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6031.852826] LustreError: 191379:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6031.886892] LustreError: 191379:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6036.964415] LustreError: 191380:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6039.631451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6040.351721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6047.288735] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6057.853796] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6065.288800] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6074.170727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6074.609766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6082.524312] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6091.459986] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6092.034690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3987 to 0x280000401:4033) [ 6098.195600] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6100.556219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6100.565546] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6100.575430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.606472] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6100.610107] LustreError: 191399:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 6104.797538] Lustre: lustre-OST0000: Not available for connect from 192.168.201.17@tcp (stopping) [ 6106.595888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6106.601020] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6106.623571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6107.620303] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6107.653673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6107.656536] Lustre: Skipped 1 previous similar message [ 6109.816392] Lustre: lustre-OST0000: Not available for connect from 192.168.201.17@tcp (stopping) [ 6114.944109] Lustre: lustre-OST0000: Not available for connect from 192.168.201.17@tcp (stopping) [ 6114.959440] Lustre: Skipped 2 previous similar messages [ 6117.346452] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6117.519621] Lustre: server umount lustre-OST0000 complete [ 6120.625995] LustreError: 191399:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 6152.683611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6153.078829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6160.270495] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6162.292056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.317450] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6167.473798] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6167.531512] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6169.420370] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6172.509073] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6179.278351] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c9453206800.ost_server_uuid 50 [ 6181.257311] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9c9453206800.ost_server_uuid in FULL state after 0 sec [ 6188.011498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6188.015161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6188.047211] Lustre: Skipped 1 previous similar message [ 6188.090789] Lustre: Skipped 1 previous similar message [ 6201.313271] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6201.827690] Lustre: server umount lustre-OST0000 complete [ 6209.515107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6209.529762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6209.575367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6209.591456] Lustre: Skipped 4 previous similar messages [ 6215.231249] Lustre: server umount lustre-MDT0000 complete [ 6219.251795] LustreError: 191380:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6219.284320] LustreError: 191380:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6220.485725] LustreError: 191360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786633319 with bad export cookie 18153961934686873697 [ 6220.495327] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6220.506117] LustreError: 191360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6220.922380] Lustre: server umount lustre-MDT0001 complete [ 6233.899895] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 6236.775611] Key type lgssc unregistered [ 6237.112497] LNet: 196608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6237.119512] LNetError: 196608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6237.146657] LNet: Removed LNI 192.168.201.117@tcp [ 6238.066898] Key type .llcrypt unregistered [ 6238.069566] Key type ._llcrypt unregistered [ 6273.973449] Key type ._llcrypt registered [ 6273.976358] Key type .llcrypt registered [ 6274.102561] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 6275.555820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6275.591967] alg: No test for adler32 (adler32-zlib) [ 6276.659453] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 6276.916980] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 6278.600211] Key type lgssc registered [ 6279.914188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6290.248947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6290.277594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6291.718893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6296.505780] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6301.172827] LustreError: 198340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6301.200341] LustreError: 198340:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6305.358727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6305.824292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6311.441267] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6315.099865] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6322.006076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6322.615580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6329.463446] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6337.395368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6337.592508] Lustre: lustre-OST0001: new disk, initializing [ 6337.597027] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6337.604342] Lustre: 200646:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6337.728818] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6338.814146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4035 to 0x280000401:4065) [ 6343.366106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6343.393471] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6343.554852] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6345.890434] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6357.586101] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6364.864414] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6377.966898] Lustre: server umount lustre-MDT0000 complete [ 6379.495023] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6379.500173] LustreError: 201426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6379.508661] Lustre: Skipped 2 previous similar messages [ 6379.531784] LustreError: 201426:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6381.713945] LustreError: 201588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786633480 with bad export cookie 11004241992425004929 [ 6381.721907] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6381.738741] LustreError: 201588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6382.140902] Lustre: server umount lustre-MDT0001 complete [ 6397.022538] Lustre: server umount lustre-OST0000 complete [ 6411.235843] Lustre: server umount lustre-OST0001 complete [ 6432.434697] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 6442.687149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6443.152229] LustreError: 203752:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6443.169801] LustreError: 203752:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6443.237666] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6448.052367] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6448.616153] LustreError: 203753:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6453.735703] LustreError: 203752:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6456.941162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6457.185519] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6461.896543] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6464.817568] Lustre: 204896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6473.909372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6474.178650] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6481.142662] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6482.421361] LustreError: 205252:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6482.443918] LustreError: 205252:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6489.656839] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4068 to 0x280000401:4097) [ 6490.682887] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6497.638779] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6505.449281] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6509.645158] Lustre: 206767:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6521.828959] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6521.831388] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6521.845601] Lustre: Skipped 3 previous similar messages [ 6521.848824] Lustre: Skipped 1 previous similar message [ 6526.951407] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6526.973549] Lustre: Skipped 3 previous similar messages [ 6528.120583] Lustre: server umount mylustre-MDT0000 complete [ 6532.066828] LustreError: 203748:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6532.113924] LustreError: 203748:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 6533.481830] LustreError: 203727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786633632 with bad export cookie 11004241992425007736 [ 6533.484407] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6533.494806] LustreError: 203727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6533.910947] Lustre: server umount mylustre-MDT0001 complete [ 6547.326178] Lustre: server umount mylustre-OST0000 complete [ 6562.313424] Lustre: server umount mylustre-OST0001 complete [ 6587.504744] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 6598.485972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6598.857614] LustreError: 209076:0:(ldlm_lib.c:1192:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6598.933933] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6598.941335] Lustre: Skipped 1 previous similar message [ 6603.699847] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6612.667151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6617.730420] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6620.834293] Lustre: 210219:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6628.157032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6635.441504] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6643.914516] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4068 to 0x280000401:4129) [ 6644.475159] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6650.393461] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6651.514321] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6659.988752] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6663.740746] Lustre: 212089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6673.819932] Lustre: server umount tfs-MDT0000 complete [ 6674.424206] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6674.435222] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6674.489453] Lustre: Skipped 1 previous similar message [ 6674.511873] LustreError: 209076:0:(ldlm_lib.c:1192:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6674.542249] LustreError: 209076:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6677.988783] LustreError: 209052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786633776 with bad export cookie 11004241992425009402 [ 6677.991325] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6678.005770] LustreError: 209052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6678.470769] Lustre: server umount tfs-MDT0001 complete [ 6692.866936] Lustre: server umount tfs-OST0000 complete [ 6708.054915] Lustre: server umount tfs-OST0001 complete [ 6731.239469] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 6742.757576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6743.336983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6743.341221] Lustre: Skipped 3 previous similar messages [ 6747.825934] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6756.588864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6761.799723] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6764.727223] Lustre: 215480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6771.630510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6778.795680] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6788.225538] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6789.642763] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6789.644968] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4068 to 0x280000401:4161) [ 6795.516240] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6804.140257] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6807.526070] Lustre: 217352:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6814.848398] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 11:11:52 (1786633912) [ 6833.633513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6833.649332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6833.675362] Lustre: Skipped 3 previous similar messages [ 6833.684107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6835.687299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6835.703941] Lustre: Skipped 2 previous similar messages [ 6837.115343] Lustre: server umount lustre-MDT0000 complete [ 6840.809415] LustreError: 214339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6840.852767] LustreError: 214339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 6840.884924] LustreError: 215081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786633939 with bad export cookie 11004241992425011068 [ 6840.890413] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6840.898957] LustreError: 215081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6845.931989] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6845.937114] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6845.946927] Lustre: Skipped 4 previous similar messages [ 6845.955476] Lustre: Skipped 2 previous similar messages [ 6847.336389] Lustre: server umount lustre-MDT0001 complete [ 6865.377078] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6865.482290] Lustre: server umount lustre-OST0000 complete [ 6875.776089] Lustre: server umount lustre-OST0001 complete [ 6883.506535] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 6895.076320] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 6935.540752] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 6935.540752] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 6935.540752] [ 6935.584746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 6935.889490] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6935.935422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6936.133407] Lustre: lustre-MDT0000: new disk, initializing [ 6936.242484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6936.249319] Lustre: Skipped 3 previous similar messages [ 6936.275702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6940.625853] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6951.958136] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 6951.958136] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 6951.958136] [ 6952.011351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 6952.117418] Lustre: 220541:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6952.167385] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6952.171654] Lustre: Skipped 1 previous similar message [ 6952.251761] Lustre: lustre-MDT0001: new disk, initializing [ 6952.403319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6952.418723] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6957.489813] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6969.633122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6970.202978] Lustre: lustre-OST0000: new disk, initializing [ 6970.210739] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6970.230746] Lustre: 222123:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6972.052050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6972.075229] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6972.163106] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6978.610933] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6990.158310] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6990.491437] Lustre: lustre-OST0001: new disk, initializing [ 6990.504141] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6990.524913] Lustre: 222995:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6992.508282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6992.520089] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6992.662644] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6998.865113] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7011.635485] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 11:15:09 (1786634109) [ 7027.242604] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 11:15:24 (1786634124) [ 7049.193534] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7049.199429] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7049.212864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7049.240243] Lustre: Skipped 1 previous similar message [ 7049.265768] Lustre: Skipped 1 previous similar message [ 7050.604437] Lustre: server umount lustre-OST0000 complete [ 7062.255788] Lustre: server umount lustre-MDT0000 complete [ 7066.112987] LustreError: 220534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786634165 with bad export cookie 11004241992425012622 [ 7066.122614] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7066.124454] LustreError: 220534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7069.678766] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7069.704848] Lustre: Skipped 3 previous similar messages [ 7069.714773] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7069.730087] Lustre: Skipped 3 previous similar messages [ 7072.662930] Lustre: server umount lustre-MDT0001 complete [ 7085.022683] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 7157.586793] Lustre: server umount lustre-OST0001 complete [ 7165.670440] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 7183.017901] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 7225.154493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7225.579785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7225.652039] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7225.791747] Lustre: lustre-MDT0000: new disk, initializing [ 7225.882349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7225.891495] Lustre: Skipped 3 previous similar messages [ 7225.916462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7231.567987] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7243.140981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7243.348896] Lustre: 227797:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7243.432325] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7243.443975] Lustre: Skipped 1 previous similar message [ 7243.522734] Lustre: lustre-MDT0001: new disk, initializing [ 7243.688633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7243.709710] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7248.893527] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7261.441614] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7267.776766] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7275.291711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7275.659792] Lustre: lustre-OST0000: new disk, initializing [ 7275.663386] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7275.670467] Lustre: 229768:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7277.432183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7277.459726] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7277.567296] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7282.785836] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7293.576544] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7303.140649] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7303.150820] LustreError: Skipped 1 previous similar message [ 7303.154690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7303.177493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7303.178651] Lustre: Skipped 1 previous similar message [ 7303.197993] Lustre: Skipped 2 previous similar messages [ 7307.005407] Lustre: server umount lustre-OST0000 complete [ 7317.337241] Lustre: server umount lustre-MDT0000 complete [ 7318.499037] LustreError: 227809:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7318.510758] LustreError: 227809:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 7321.140342] LustreError: 227791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786634420 with bad export cookie 11004241992425014092 [ 7321.147990] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7321.161738] LustreError: 227791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7321.699955] Lustre: server umount lustre-MDT0001 complete [ 7334.149628] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 7336.973926] Key type lgssc unregistered [ 7337.232692] LNet: 231532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7337.239427] LNetError: 231532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7337.252070] LNet: Removed LNI 192.168.201.117@tcp [ 7338.145177] Key type .llcrypt unregistered [ 7338.147236] Key type ._llcrypt unregistered [ 7367.131812] Key type ._llcrypt registered [ 7367.141777] Key type .llcrypt registered [ 7367.304248] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 7367.970461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7368.012561] alg: No test for adler32 (adler32-zlib) [ 7369.256344] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 7369.573374] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 7371.296619] Key type lgssc registered [ 7372.877474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7381.396734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7381.416166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7383.114106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7388.724122] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7393.257833] LustreError: 232995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7393.289871] LustreError: 232995:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7398.380103] LustreError: 232996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7398.597416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7399.159236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7405.069944] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7415.753938] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7423.134879] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7430.870392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7431.310709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7437.641904] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7445.434518] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7446.449617] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 7450.091623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7450.109107] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7450.126885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7450.634989] Lustre: server umount lustre-OST0000 complete [ 7455.714454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7455.728275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7455.750704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7459.297918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7459.301070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7461.816270] Lustre: server umount lustre-MDT0000 complete [ 7464.417452] LustreError: 232996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7464.435390] LustreError: 232996:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7466.861146] LustreError: 232976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786634565 with bad export cookie 12495453905909240091 [ 7466.863521] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7466.869784] LustreError: 232976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7467.439349] Lustre: server umount lustre-MDT0001 complete [ 7480.883169] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 7484.443899] Key type lgssc unregistered [ 7484.865545] LNet: 236583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7484.878493] LNetError: 236583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7485.926829] LNet: Removed LNI 192.168.201.117@tcp [ 7486.811613] Key type .llcrypt unregistered [ 7486.814793] Key type ._llcrypt unregistered [ 7504.733360] Key type ._llcrypt registered [ 7504.738943] Key type .llcrypt registered [ 7504.866944] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 7505.840618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7505.873445] alg: No test for adler32 (adler32-zlib) [ 7506.903675] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 7507.174781] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 7508.848257] Key type lgssc registered [ 7510.784644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7521.937275] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7521.961270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7537.696414] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7542.816819] LustreError: 237778:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.117@tcp: failed processing log, type 4: rc = -110 [ 7569.504509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7576.556459] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7584.526627] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7587.147900] Lustre: server umount lustre-OST0000 complete [ 7604.287197] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 7606.819529] Key type lgssc unregistered [ 7607.073814] LNet: 239413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7607.081300] LNetError: 239413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7608.107418] LNet: Removed LNI 192.168.201.117@tcp [ 7609.081682] Key type .llcrypt unregistered [ 7609.093359] Key type ._llcrypt unregistered [ 7650.183253] Key type ._llcrypt registered [ 7650.184770] Key type .llcrypt registered [ 7650.277738] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 7651.151376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7651.258064] alg: No test for adler32 (adler32-zlib) [ 7652.286769] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 7652.507771] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 7654.176551] Key type lgssc registered [ 7655.684891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7669.608791] loop0: detected capacity change from 0 to 204800000 [ 7669.849783] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7669.868791] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7670.345293] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7671.508552] loop0: detected capacity change from 0 to 204800000 [ 7671.789485] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7671.795841] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7672.170532] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7673.182055] loop0: detected capacity change from 0 to 204800000 [ 7673.368660] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7673.383427] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7673.548221] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7674.865529] loop0: detected capacity change from 0 to 204800000 [ 7675.024413] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7675.034456] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7675.171104] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7676.362516] loop0: detected capacity change from 0 to 204800000 [ 7676.404232] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 7681.263408] loop0: detected capacity change from 0 to 204800000 [ 7681.298283] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 7685.560343] loop0: detected capacity change from 0 to 204800000 [ 7685.615476] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 7690.145927] loop0: detected capacity change from 0 to 204800000 [ 7690.186947] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 7695.046812] loop0: detected capacity change from 0 to 204800000 [ 7695.201303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7695.266385] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7695.345383] Lustre: 242828:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 7695.356660] Lustre: 242828:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 7695.370787] Lustre: 242828:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 7695.382850] Lustre: 242828:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 7695.393859] Lustre: 242828:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 7695.409761] Lustre: 242828:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 7695.422464] Lustre: 242828:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 7695.433632] Lustre: 242828:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 7695.445564] Lustre: 242828:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 7695.463117] Lustre: 242828:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 7695.478136] Lustre: 242828:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 7695.492273] Lustre: 242828:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 7695.504698] Lustre: 242828:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 7695.514793] Lustre: 242828:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 7702.803273] Lustre: server umount MGS complete [ 7703.983957] loop0: detected capacity change from 0 to 204800000 [ 7704.139265] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7704.802463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7704.855874] Lustre: lustre-MDT0000: Aborting client recovery [ 7706.231262] loop1: detected capacity change from 0 to 204800000 [ 7706.415463] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7706.544191] Lustre: 243263:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 7706.570614] Lustre: 243263:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 7706.593663] Lustre: 243263:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 7706.613506] Lustre: 243263:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 7706.637661] Lustre: 243263:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 7706.668931] Lustre: 243263:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 7706.699146] Lustre: 243263:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 7706.743740] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7706.934939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7707.008093] Lustre: lustre-MDT0001: Aborting client recovery [ 7708.201978] loop2: detected capacity change from 0 to 204800000 [ 7708.520886] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7708.621591] Lustre: 243341:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 7708.644869] Lustre: 243341:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 7708.655856] Lustre: 243341:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 7708.669916] Lustre: 243341:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 7708.688596] Lustre: 243341:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 7708.701686] Lustre: 243341:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 7708.720889] Lustre: 243341:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 7708.981493] Lustre: Found index 0 for lustre-OST0000, updating log [ 7709.039458] Lustre: 243359:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7709.131568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7709.144772] Lustre: lustre-OST0000: Aborting recovery [ 7710.198441] loop3: detected capacity change from 0 to 204800000 [ 7710.427061] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7710.573426] Lustre: 243424:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 7710.591616] Lustre: 243424:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 7710.607772] Lustre: 243424:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 7710.621705] Lustre: 243424:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 7710.643409] Lustre: 243424:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 7710.655365] Lustre: 243424:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 7710.677209] Lustre: 243424:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 7710.730960] Lustre: Found index 1 for lustre-OST0001, updating log [ 7710.813629] Lustre: 243434:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7710.959280] Lustre: lustre-OST0001: Aborting recovery [ 7714.052293] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 7714.104267] LustreError: 243206:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 7714.175121] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 7714.198547] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 7714.293101] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 7716.074869] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 7716.075553] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 7720.221959] LustreError: 243173:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 796us: rc = -2 [ 7720.749967] LustreError: 243174:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0001: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0001 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0001/identity_upcall, time 2226us: rc = -2 [ 7720.777935] LustreError: 243174:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 14 previous similar messages [ 7736.802970] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7736.806773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7736.826568] Lustre: Skipped 1 previous similar message [ 7740.095336] Lustre: server umount lustre-MDT0000 complete [ 7741.056257] LustreError: 243155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786634840 with bad export cookie 10034756830137452174 [ 7741.118641] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7741.921027] LustreError: 243172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7741.921144] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7741.923140] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7741.923146] Lustre: Skipped 3 previous similar messages [ 7741.929893] LustreError: 243172:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7741.975937] Lustre: Skipped 3 previous similar messages [ 7747.044501] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7747.045453] LustreError: 243667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7747.057972] Lustre: Skipped 2 previous similar messages [ 7747.103989] LustreError: 243667:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7747.728354] Lustre: server umount lustre-MDT0001 complete [ 7755.179252] Lustre: server umount lustre-OST0000 complete [ 7762.674592] Lustre: server umount lustre-OST0001 complete [ 7770.749439] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 11:27:48 (1786634868) [ 7802.732772] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 7813.284382] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 7853.013345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7853.403277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7853.434426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7853.523747] Lustre: lustre-MDT0000: new disk, initializing [ 7853.622146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7853.627734] Lustre: Skipped 1 previous similar message [ 7853.647507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7859.009989] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7871.069466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7871.267696] Lustre: 247504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7871.413561] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7871.427456] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7871.445351] Lustre: Skipped 1 previous similar message [ 7871.556047] Lustre: lustre-MDT0001: new disk, initializing [ 7871.669067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7871.717380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7871.742407] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7876.795507] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7887.622705] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7894.129470] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7901.771701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7902.018894] Lustre: lustre-OST0000: new disk, initializing [ 7902.033746] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7902.038764] Lustre: 249469:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7902.122116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7907.455581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7907.475341] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7907.641611] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7909.756544] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7920.888268] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7926.316479] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 7938.263415] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 7944.485327] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 7950.222400] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 7999.969563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7999.975880] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7999.998175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7999.998783] Lustre: Skipped 1 previous similar message [ 8000.017077] Lustre: Skipped 1 previous similar message [ 8005.015130] Lustre: server umount lustre-OST0000 complete [ 8010.209161] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8010.209265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8010.214458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8010.214967] LustreError: Skipped 1 previous similar message [ 8010.230956] Lustre: Skipped 1 previous similar message [ 8015.334152] LustreError: 247515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8015.558821] Lustre: server umount lustre-MDT0000 complete [ 8020.458152] LustreError: 250989:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8020.487155] LustreError: 250989:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8020.631229] LustreError: 247496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786635119 with bad export cookie 10034756830137454946 [ 8020.637899] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8020.647910] LustreError: 247496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8021.329608] Lustre: server umount lustre-MDT0001 complete [ 8031.598093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8037.687828] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8043.899485] Lustre: server umount MGS complete [ 8055.806815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8056.239415] LustreError: 253338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8056.267801] LustreError: 253351:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 8056.325376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8061.496694] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8066.534376] LustreError: 253339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8066.554324] LustreError: 253339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8070.911988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8076.245115] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8084.907767] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8090.744585] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8097.723422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8098.283221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8098.292193] Lustre: Skipped 1 previous similar message [ 8105.470752] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8114.872563] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8147.432353] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8147.434066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8147.442304] LustreError: Skipped 1 previous similar message [ 8147.443394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8147.480785] Lustre: Skipped 2 previous similar messages [ 8154.156293] Lustre: server umount lustre-OST0000 complete [ 8162.788112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8162.804130] Lustre: Skipped 1 previous similar message [ 8165.315538] Lustre: server umount lustre-MDT0000 complete [ 8167.911582] LustreError: 255179:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8167.939146] LustreError: 255179:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8169.495724] LustreError: 253318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786635268 with bad export cookie 10034756830137456808 [ 8169.507542] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8169.946473] Lustre: server umount lustre-MDT0001 complete [ 8182.607841] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 8186.213634] Key type lgssc unregistered [ 8186.593807] LNet: 257290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8186.619280] LNetError: 257290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8186.654226] LNet: Removed LNI 192.168.201.117@tcp [ 8187.866476] Key type .llcrypt unregistered [ 8187.872229] Key type ._llcrypt unregistered [ 8230.197638] Key type ._llcrypt registered [ 8230.202793] Key type .llcrypt registered [ 8230.380154] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 8247.644512] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 8248.906542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8248.924286] alg: No test for adler32 (adler32-zlib) [ 8250.155882] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 8250.551488] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 8252.353838] Key type lgssc registered [ 8253.928440] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8294.088613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8294.123651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8295.538164] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8295.589384] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8295.729422] Lustre: lustre-MDT0000: new disk, initializing [ 8295.980199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8296.016942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8300.895311] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8312.636975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8312.809063] Lustre: 261250:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8312.888226] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8312.898956] Lustre: Skipped 1 previous similar message [ 8313.040108] Lustre: lustre-MDT0001: new disk, initializing [ 8313.180678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8313.241706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8313.278708] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8319.190865] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8329.969557] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8336.119868] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8343.714734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8343.991908] Lustre: lustre-OST0000: new disk, initializing [ 8343.994978] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8344.012971] Lustre: 263218:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8344.162942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8350.585417] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8351.316821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8351.344368] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8351.449845] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8360.723237] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8368.653678] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8382.172468] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8386.231524] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 8433.658789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8433.664547] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8433.670399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8433.686317] Lustre: Skipped 1 previous similar message [ 8433.702217] LustreError: Skipped 1 previous similar message [ 8438.768786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8438.776093] Lustre: Skipped 2 previous similar messages [ 8439.687159] Lustre: server umount lustre-OST0000 complete [ 8446.434492] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8446.451378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8446.475555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8448.994177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8449.002375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8450.489505] Lustre: server umount lustre-MDT0000 complete [ 8453.479222] LustreError: 261243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786635552 with bad export cookie 8064481107997615889 [ 8453.481793] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8453.487357] LustreError: 261243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8453.797586] Lustre: server umount lustre-MDT0001 complete [ 8463.653483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8470.177553] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8475.407421] Lustre: server umount MGS complete [ 8490.368379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8490.968679] LustreError: 267097:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 8491.055680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8496.097894] LustreError: 267084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8496.121283] LustreError: 267084:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8496.976397] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8500.200955] LustreError: 267085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8505.312984] LustreError: 267084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8506.152693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8506.462171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8513.537452] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8522.431306] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8529.200561] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8536.445627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8536.821193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8543.838722] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8551.908581] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8596.452044] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8596.470516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8596.479987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8596.480026] Lustre: Skipped 1 previous similar message [ 8596.527732] Lustre: Skipped 1 previous similar message [ 8606.694270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8606.711466] Lustre: Skipped 4 previous similar messages [ 8609.248291] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8609.686651] Lustre: server umount lustre-OST0000 complete [ 8616.948098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8616.976563] Lustre: Skipped 1 previous similar message [ 8627.170309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8627.180092] Lustre: Skipped 5 previous similar messages [ 8628.192126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8628.797895] Lustre: server umount lustre-MDT0000 complete [ 8632.324719] LustreError: 267993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8632.340730] LustreError: 267993:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8632.934816] LustreError: 267821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786635731 with bad export cookie 8064481107997617730 [ 8632.938742] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8632.951614] LustreError: 267821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8633.491806] Lustre: server umount lustre-MDT0001 complete [ 8645.773684] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 8649.114301] Key type lgssc unregistered [ 8649.544743] LNet: 271061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8649.566994] LNetError: 271061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8649.588382] LNet: Removed LNI 192.168.201.117@tcp [ 8650.474843] Key type .llcrypt unregistered [ 8650.479420] Key type ._llcrypt unregistered [ 8671.831982] Key type ._llcrypt registered [ 8671.840658] Key type .llcrypt registered [ 8671.943348] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 8686.558919] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 8688.240687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8688.275115] alg: No test for adler32 (adler32-zlib) [ 8689.503934] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 8689.700781] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 8691.456194] Key type lgssc registered [ 8692.694733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8717.917584] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 8734.306875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8734.345788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8735.713367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8735.739228] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8735.820896] Lustre: lustre-MDT0000: new disk, initializing [ 8735.903262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8735.926824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8741.442539] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8752.578328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8753.062586] Lustre: lustre-OST0000: new disk, initializing [ 8753.068163] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 8753.071338] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8753.071348] Lustre: Skipped 1 previous similar message [ 8753.081037] Lustre: 275113:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8753.187144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8758.295926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8758.303717] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8758.343371] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8760.432954] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9563.108586] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9563.129401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9564.392300] Lustre: server umount lustre-MDT0000 complete [ 9568.562636] LustreError: 274297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786636667 with bad export cookie 12486255906412547078 [ 9568.575133] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9568.590394] LustreError: 274297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9568.974919] Lustre: server umount lustre-OST0000 complete [ 9599.958864] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 9610.708098] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 9648.349831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9648.582310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9648.609180] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9648.707898] Lustre: lustre-MDT0000: new disk, initializing [ 9648.786227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9648.798707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9652.959466] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9663.634640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9663.774476] Lustre: 279439:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9663.816275] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9663.819476] Lustre: Skipped 1 previous similar message [ 9663.893492] Lustre: lustre-MDT0001: new disk, initializing [ 9663.995392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9664.073108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9664.093140] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9668.843156] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9679.606119] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9685.910253] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9692.821065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9693.200659] Lustre: lustre-OST0000: new disk, initializing [ 9693.206536] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9693.212708] Lustre: 281406:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9693.338106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9700.433452] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9701.981249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9702.004435] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9702.084460] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9709.864557] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9714.942390] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9715.287723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9720.013603] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9720.295253] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9722.853850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9722.868405] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9722.886781] Lustre: Skipped 1 previous similar message [ 9722.887706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9722.919313] LustreError: Skipped 1 previous similar message [ 9727.975269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9727.982094] Lustre: Skipped 2 previous similar messages [ 9728.997967] Lustre: server umount lustre-OST0000 complete [ 9735.666952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9735.677200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9735.691116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9738.209671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9739.974230] Lustre: server umount lustre-MDT0000 complete [ 9744.399080] LustreError: 281405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786636843 with bad export cookie 12486255906415487890 [ 9744.400084] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9744.410885] LustreError: 281405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9744.764704] Lustre: server umount lustre-MDT0001 complete [ 9748.596914] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [ 9750.262755] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 12:00:48 (1786636848) [ 9758.030044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9758.592540] LustreError: 283719:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9758.615238] LustreError: 283719:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9758.756785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9763.812873] LustreError: 283720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9764.456725] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9768.936437] LustreError: 283719:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9774.053126] LustreError: 283720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9775.417092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9775.939111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9780.371698] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9789.426057] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9797.289990] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9803.990718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9804.450719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9811.556608] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9819.907923] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9827.636986] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9827.837241] Lustre: lustre-OST0001: new disk, initializing [ 9827.840759] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9827.846354] Lustre: 286568:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9827.935096] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9835.223192] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9836.692097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9836.713126] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9836.760535] LustreError: 287001:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 9836.784277] LustreError: 287001:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 9837.797600] LustreError: 287006:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 9837.808936] LustreError: 287006:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 9838.829052] LustreError: 287001:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 9840.867826] LustreError: 287001:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 9840.889937] LustreError: 287001:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 9845.027421] LustreError: 287001:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 9845.051182] LustreError: 287001:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [ 9845.555520] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9845.781499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9852.085659] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c9448090000.ost_server_uuid 50 [ 9853.353746] LustreError: 287001:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 9853.367702] LustreError: 287001:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [ 9853.915266] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9c9448090000.ost_server_uuid in FULL state after 0 sec [ 9860.731664] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c9448090000.ost_server_uuid 50 [ 9862.384511] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9c9448090000.ost_server_uuid in IDLE state after 0 sec [ 9864.779415] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9880.033627] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9880.041892] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9880.059125] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9880.063671] Lustre: Skipped 2 previous similar messages [ 9884.501039] Lustre: server umount lustre-OST0001 complete [ 9886.323371] LustreError: 285553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9892.835031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9892.843597] LustreError: Skipped 1 previous similar message [ 9892.851568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9892.858253] Lustre: Skipped 1 previous similar message [ 9892.862032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9892.864715] Lustre: Skipped 4 previous similar messages [ 9897.890297] Lustre: server umount lustre-OST0000 complete [ 9909.085365] Lustre: server umount lustre-MDT0000 complete [ 9912.626883] LustreError: 283701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637011 with bad export cookie 12486255906415488779 [ 9912.634877] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9912.647387] LustreError: 283701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9912.996784] Lustre: server umount lustre-MDT0001 complete [ 9924.942752] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 9928.273729] Key type lgssc unregistered [ 9928.532554] LNet: 289262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9928.545656] LNetError: 289262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9929.575981] LNet: Removed LNI 192.168.201.117@tcp [ 9930.656355] Key type .llcrypt unregistered [ 9930.660119] Key type ._llcrypt unregistered [ 9955.144566] Key type ._llcrypt registered [ 9955.147246] Key type .llcrypt registered [ 9955.235772] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 9956.289673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9956.427392] alg: No test for adler32 (adler32-zlib) [ 9957.451292] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 9957.702623] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 9959.361622] Key type lgssc registered [ 9960.948652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9970.415813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9970.455306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9971.853119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9977.733753] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9981.432292] LustreError: 290686:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9981.461950] LustreError: 290686:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9985.586898] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9986.532805] LustreError: 290685:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9991.655310] LustreError: 290686:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9992.710141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9993.359405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9998.741709] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10009.373272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10010.024395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10016.191961] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [10016.353142] LustreError: 292367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10017.652782] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10023.430915] LustreError: 292665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10023.470621] LustreError: 292665:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10025.113308] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10032.330145] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10032.644379] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10037.781130] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [10039.056502] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10045.702887] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10052.540383] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [10052.721827] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10150.375292] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10150.376415] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10150.382499] LustreError: Skipped 1 previous similar message [10150.383518] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10150.389277] Lustre: Skipped 1 previous similar message [10155.491903] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10155.503126] Lustre: Skipped 2 previous similar messages [10160.619800] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10160.627375] Lustre: Skipped 1 previous similar message [10164.192982] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10164.299592] Lustre: server umount lustre-OST0001 complete [10165.739662] LustreError: 292665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10165.748626] LustreError: 292665:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10170.853683] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10170.874731] Lustre: Skipped 1 previous similar message [10170.875549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10170.886434] Lustre: Skipped 1 previous similar message [10175.971289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10175.982428] Lustre: Skipped 1 previous similar message [10184.160879] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10184.371149] Lustre: server umount lustre-OST0000 complete [10187.745767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10187.751518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10187.771520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10187.775859] Lustre: Skipped 2 previous similar messages [10191.335486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10193.985855] Lustre: server umount lustre-MDT0000 complete [10196.449456] LustreError: 290680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10196.477276] LustreError: 290680:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10197.787791] LustreError: 294348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637296 with bad export cookie 7871554335089144891 [10197.795724] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10197.806885] LustreError: 294348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10198.255816] Lustre: server umount lustre-MDT0001 complete [10207.232414] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [10209.707822] Key type lgssc unregistered [10209.948702] LNet: 295633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10209.952832] LNetError: 295633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10209.967098] LNet: Removed LNI 192.168.201.117@tcp [10210.745452] Key type .llcrypt unregistered [10210.748895] Key type ._llcrypt unregistered [10254.853588] Key type ._llcrypt registered [10254.856981] Key type .llcrypt registered [10254.946978] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [10256.042584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10256.052678] alg: No test for adler32 (adler32-zlib) [10257.098204] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [10257.321099] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [10258.992642] Key type lgssc registered [10259.901706] Lustre: Echo OBD driver; http://www.lustre.org/ [10268.392120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10268.406735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10269.755909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10274.345913] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10279.905347] LustreError: 297924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10279.935052] LustreError: 297924:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10283.393851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10283.726914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10287.988335] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10290.609789] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10297.517085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10297.957184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10304.254564] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10308.195588] LustreError: 299954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10312.928300] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10313.167366] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10314.258645] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [10314.280243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [10319.758528] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10327.852383] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10333.810544] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10350.058187] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10350.068365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10350.076715] Lustre: Skipped 2 previous similar messages [10350.089582] Lustre: Skipped 3 previous similar messages [10351.876134] Lustre: server umount lustre-MDT0000 complete [10355.185562] LustreError: 297925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10355.207060] LustreError: 297925:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10355.338783] LustreError: 297905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637454 with bad export cookie 441733278154488392 [10355.346766] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10355.349276] LustreError: 297905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10355.846750] Lustre: server umount lustre-MDT0001 complete [10369.658324] Lustre: server umount lustre-OST0000 complete [10382.358896] Lustre: server umount lustre-OST0001 complete [10401.365836] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [10411.392444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10411.791753] LustreError: 303187:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10411.853102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10416.992852] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10417.127619] LustreError: 303188:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10426.103311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10426.338960] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10426.606717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10432.366643] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10435.974650] Lustre: 304327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10443.095086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10443.634220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10451.303675] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10451.824696] LustreError: 304680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10451.842896] LustreError: 304680:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10456.960913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [10460.952850] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10466.814815] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [10468.448495] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10476.536324] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10480.042683] Lustre: 306201:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10487.271846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10487.275923] Lustre: Skipped 2 previous similar messages [10487.278276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10487.285249] Lustre: Skipped 3 previous similar messages [10490.526574] Lustre: server umount lustre-MDT0000 complete [10492.388241] LustreError: 303187:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10492.417578] LustreError: 303187:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10493.988735] LustreError: 303167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637592 with bad export cookie 441733278154490086 [10494.004592] LustreError: 303167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10494.007241] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10494.381775] Lustre: server umount lustre-MDT0001 complete [10509.327701] Lustre: server umount lustre-OST0000 complete [10513.056307] Lustre: 297186:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786637596/real 1786637596] req@ffff8a8d08c7ed80 x1873425052317184/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786637612 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10513.104815] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10513.123093] Lustre: Skipped 2 previous similar messages [10513.475869] Lustre: server umount lustre-OST0001 complete [10516.700086] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [10517.949956] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [10519.711543] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 12:13:37 (1786637617) [10543.198541] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [10552.987771] loop0: detected capacity change from 0 to 1099511627776 [10553.823201] SGI XFS with ACLs, security attributes, quota, no debug enabled [10553.835504] *** VALIDATE xfs *** [10553.871091] XFS (loop0): Mounting V5 Filesystem [10553.932202] XFS (loop0): Ending clean mount [10555.274451] loop1: detected capacity change from 0 to 18691697672192 [10675.738073] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [10685.914901] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [10723.598605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10723.874753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10723.908918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10724.039757] Lustre: lustre-MDT0000: new disk, initializing [10724.196052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10724.205745] Lustre: Skipped 1 previous similar message [10724.254703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10729.475680] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10740.814282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10740.974183] Lustre: 311343:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10741.070146] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10741.078239] Lustre: Skipped 1 previous similar message [10741.200467] Lustre: lustre-MDT0001: new disk, initializing [10741.428732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10741.445679] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10747.800930] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10759.851909] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10766.987756] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10773.538183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10773.932536] Lustre: lustre-OST0000: new disk, initializing [10773.937709] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10773.951907] Lustre: 313309:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10775.488848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10775.507822] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10775.650764] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10780.300873] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10789.669550] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10794.585736] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10794.828667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10798.885941] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10799.058677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10801.121390] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10801.130971] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10801.131420] LustreError: Skipped 1 previous similar message [10801.135096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10801.142215] Lustre: Skipped 1 previous similar message [10801.162693] Lustre: Skipped 1 previous similar message [10806.247502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10806.258351] Lustre: Skipped 1 previous similar message [10807.169852] Lustre: server umount lustre-OST0000 complete [10811.362183] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10811.376556] Lustre: Skipped 1 previous similar message [10816.484337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10816.491977] Lustre: Skipped 3 previous similar messages [10817.656768] Lustre: server umount lustre-MDT0000 complete [10821.157060] LustreError: 313308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637920 with bad export cookie 441733278154491647 [10821.164425] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10821.165894] LustreError: 313308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10821.440635] Lustre: server umount lustre-MDT0001 complete [10828.629181] XFS (loop0): Unmounting Filesystem [10832.732022] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 12:18:50 (1786637930) [10839.937936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10840.740475] LustreError: 315867:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10840.890025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10840.896011] Lustre: Skipped 2 previous similar messages [10845.373283] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10854.837158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10861.024340] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10870.013277] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10876.509127] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10883.745426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10890.796662] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10898.318308] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10910.695391] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10910.710882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10910.740430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10916.984065] Lustre: server umount lustre-OST0000 complete [10921.442681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10927.306791] Lustre: server umount lustre-MDT0000 complete [10930.658868] LustreError: 316596:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10930.669829] LustreError: 316596:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10931.375119] LustreError: 315848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786638030 with bad export cookie 441733278154492529 [10931.383716] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10931.388502] LustreError: 315848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10931.812635] Lustre: server umount lustre-MDT0001 complete [10944.333305] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [10947.672837] Key type lgssc unregistered [10947.880742] LNet: 319500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10947.889607] LNetError: 319500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10948.969170] LNet: Removed LNI 192.168.201.117@tcp [10950.004138] Key type .llcrypt unregistered [10950.006477] Key type ._llcrypt unregistered [10980.783353] Key type ._llcrypt registered [10980.789838] Key type .llcrypt registered [10980.966979] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [10982.071179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10982.105214] alg: No test for adler32 (adler32-zlib) [10983.675174] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [10984.017672] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [10985.704946] Key type lgssc registered [10986.592430] Lustre: Echo OBD driver; http://www.lustre.org/ [10994.847298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10994.866956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10996.369338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11003.128566] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11006.954195] LustreError: 321069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11006.977612] LustreError: 321069:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11011.850751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11012.076231] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11012.230562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11016.824649] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11027.426376] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11036.186396] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11043.572592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11044.115261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11051.310720] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11057.757867] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11077.099560] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11077.113028] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11078.798359] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.17@tcp (stopping) [11078.807525] Lustre: Skipped 1 previous similar message [11080.783347] Lustre: server umount lustre-MDT0001 complete [11082.213555] LustreError: 321064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11082.249551] LustreError: 321064:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11083.906697] LustreError: 321065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11087.338050] LustreError: 323563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11087.384719] LustreError: 323563:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11092.452521] LustreError: 321066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11092.468375] LustreError: 321066:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11093.687675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11093.806474] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11093.810572] Lustre: Found index 1 for lustre-MDT0001, updating log [11093.906104] Lustre: 321057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11094.290728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11098.417748] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11099.661547] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11101.189393] LustreError: 324668:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11101.193536] LustreError: 324668:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11101.200840] LustreError: 321062:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11101.237088] LustreError: 324671:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.117@tcp: cfg command failed: rc = -17 [11101.244130] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [11101.244130] [11109.860474] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11109.862838] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11109.891591] Lustre: Skipped 1 previous similar message [11109.917131] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11114.639927] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.17@tcp (stopping) [11114.657448] Lustre: Skipped 1 previous similar message [11116.092730] Lustre: server umount lustre-MDT0001 complete [11119.730883] LustreError: 321066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11127.702976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11127.934830] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11127.943523] Lustre: Found index 1 for lustre-MDT0001, updating log [11128.019874] Lustre: 321059:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11128.382591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11133.446509] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11133.591575] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11135.019689] LustreError: 325688:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11135.040264] LustreError: 325688:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11135.052284] LustreError: 321062:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11135.074323] LustreError: 321062:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [11135.212645] LustreError: 325695:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [11135.232527] LustreError: 325695:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [11135.245896] LustreError: 325695:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.117@tcp: cfg command failed: rc = -17 [11135.260770] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.201.117@tcp [11135.260770] [11440.286968] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.17@tcp (stopping) [11440.293812] Lustre: Skipped 2 previous similar messages [11440.613422] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11440.615989] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11446.184190] Lustre: server umount lustre-MDT0001 complete [11450.491319] LustreError: 321065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11450.514395] LustreError: 321065:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11455.949521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11456.083870] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11456.090660] Lustre: Found index 1 for lustre-MDT0001, updating log [11456.161652] Lustre: 321057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11456.611071] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [11461.627178] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11461.651815] LustreError: 333934:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11461.657427] LustreError: 333934:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11461.664250] LustreError: 321062:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11461.676814] LustreError: 321062:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [11461.756793] LustreError: 333947:0:(lod_lov.c:210:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [11461.780136] LustreError: 333947:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.117@tcp: cfg command failed: rc = -114 [11461.789136] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [11461.789136] [11461.831636] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11784.168274] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11784.179031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11784.179043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11784.179052] Lustre: Skipped 5 previous similar messages [11784.187886] Lustre: Skipped 2 previous similar messages [11788.827224] Lustre: server umount lustre-MDT0000 complete [11789.283128] LustreError: 322078:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11789.326582] LustreError: 322078:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [11793.307514] LustreError: 321050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786638892 with bad export cookie 1553319238908841098 [11793.325334] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11793.326040] LustreError: 321050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11794.402649] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11794.421313] Lustre: Skipped 1 previous similar message [11800.044928] Lustre: server umount lustre-MDT0001 complete [11804.004606] Lustre: server umount lustre-OST0000 complete [11813.805487] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [11824.635397] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [11860.562557] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 12:35:58 (1786638958) [11867.318089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11867.517456] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11867.552542] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11867.657556] Lustre: lustre-MDT0000: new disk, initializing [11867.735901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11867.755468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11871.793466] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11882.635413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11882.817150] Lustre: 345081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11882.902400] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11882.916975] Lustre: Skipped 1 previous similar message [11883.013249] Lustre: lustre-MDT0001: new disk, initializing [11883.129529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11883.186191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11883.194667] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11887.940276] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11898.448506] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11905.622341] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11914.081973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11914.385821] Lustre: lustre-OST0000: new disk, initializing [11914.389101] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11914.399144] Lustre: 347050:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11914.508484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11921.889642] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11924.554976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11924.560592] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11924.700648] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11932.409569] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11940.327375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11940.348453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11940.364021] Lustre: Skipped 2 previous similar messages [11940.371702] Lustre: Skipped 5 previous similar messages [11943.820541] Lustre: server umount lustre-MDT0000 complete [11945.446283] LustreError: 346162:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11945.469918] LustreError: 346162:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11948.027897] LustreError: 345075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786639047 with bad export cookie 1553319238908843051 [11948.029586] LustreError: 345887:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [11948.041222] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11948.053170] LustreError: 345075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11948.559350] Lustre: server umount lustre-MDT0001 complete [11963.150974] Lustre: server umount lustre-OST0000 complete [11991.948870] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [12002.658395] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [12037.842423] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11710 sec ========= 12:38:55 (1786639135) [12039.683297] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:38:57 (1786639137) === [12043.430917] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:39:01 (1786639141) === [12088.624402] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [12092.631982] Key type lgssc unregistered [12093.047645] LNet: 353419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12093.055883] LNetError: 353419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12094.123936] LNet: Removed LNI 192.168.201.117@tcp [12095.318734] Key type .llcrypt unregistered [12095.324988] Key type ._llcrypt unregistered