[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 864623340 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2672MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003134] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.009514] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011014] pid_max: default: 32768 minimum: 301 [ 0.012146] LSM: Security Framework initializing [ 0.013048] Yama: becoming mindful. [ 0.014056] SELinux: Initializing. [ 0.016026] *** VALIDATE selinux *** [ 0.031460] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.040156] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.041145] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.042104] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.043133] *** VALIDATE tmpfs *** [ 0.046388] *** VALIDATE proc *** [ 0.048154] *** VALIDATE cgroup *** [ 0.049016] *** VALIDATE cgroup2 *** [ 0.051133] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.052165] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.054016] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.055030] Spectre V2 : User space: Vulnerable [ 0.056013] Speculative Store Bypass: Vulnerable [ 0.061887] debug: unmapping init [mem 0xffffffffba859000-0xffffffffba860fff] [ 0.064301] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.068105] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.069033] ... version: 2 [ 0.070017] ... bit width: 48 [ 0.071017] ... generic registers: 4 [ 0.072019] ... value mask: 0000ffffffffffff [ 0.073014] ... max period: 00007fffffffffff [ 0.074017] ... fixed-purpose events: 3 [ 0.075015] ... event mask: 000000070000000f [ 0.076275] rcu: Hierarchical SRCU implementation. [ 0.079554] smp: Bringing up secondary CPUs ... [ 0.082034] x86: Booting SMP configuration: [ 0.083018] .... node #0, CPUs: #1 #2 #3 [ 0.096171] smp: Brought up 1 node, 4 CPUs [ 0.098015] smpboot: Max logical packages: 1 [ 0.099017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.148954] node 0 deferred pages initialised in 45ms [ 0.154553] devtmpfs: initialized [ 0.158337] x86/mm: Memory block size: 128MB [ 0.162633] gcov: version magic: 0x41383552 [ 0.165538] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.170091] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.174271] pinctrl core: initialized pinctrl subsystem [ 0.179652] [ 0.180012] ************************************************************* [ 0.187015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.193012] ** ** [ 0.202015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.207021] ** ** [ 0.210011] ** This means that this kernel is built to expose internal ** [ 0.214013] ** IOMMU data structures, which may compromise security on ** [ 0.218012] ** your system. ** [ 0.224015] ** ** [ 0.228012] ** If you see this message and you are not debugging the ** [ 0.232014] ** kernel, report this immediately to your vendor! ** [ 0.237014] ** ** [ 0.240016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.245014] ************************************************************* [ 0.254152] NET: Registered protocol family 16 [ 0.259079] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.265060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.270062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.278140] cpuidle: using governor menu [ 0.283970] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.288888] PCI: Using configuration type 1 for base access [ 0.294138] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.304078] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.307097] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.318143] cryptd: max_cpu_qlen set to 1000 [ 0.327074] ACPI: Added _OSI(Module Device) [ 0.332019] ACPI: Added _OSI(Processor Device) [ 0.336015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.341016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.362032] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.379701] ACPI: Interpreter enabled [ 0.383077] ACPI: PM: (supports S0 S3 S4 S5) [ 0.386128] ACPI: Using IOAPIC for interrupt routing [ 0.389385] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.399116] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.416250] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.420041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.423019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.429078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.439397] acpiphp: Slot [2] registered [ 0.441212] acpiphp: Slot [5] registered [ 0.444196] acpiphp: Slot [6] registered [ 0.445116] acpiphp: Slot [7] registered [ 0.448181] acpiphp: Slot [8] registered [ 0.450332] acpiphp: Slot [9] registered [ 0.453117] acpiphp: Slot [10] registered [ 0.455108] acpiphp: Slot [3] registered [ 0.457079] acpiphp: Slot [4] registered [ 0.458096] acpiphp: Slot [11] registered [ 0.459104] acpiphp: Slot [12] registered [ 0.461117] acpiphp: Slot [13] registered [ 0.463088] acpiphp: Slot [14] registered [ 0.464083] acpiphp: Slot [15] registered [ 0.466135] acpiphp: Slot [16] registered [ 0.468090] acpiphp: Slot [17] registered [ 0.470163] acpiphp: Slot [18] registered [ 0.472597] acpiphp: Slot [19] registered [ 0.475227] acpiphp: Slot [20] registered [ 0.479114] acpiphp: Slot [21] registered [ 0.483679] acpiphp: Slot [22] registered [ 0.488656] acpiphp: Slot [23] registered [ 0.491477] acpiphp: Slot [24] registered [ 0.494750] acpiphp: Slot [25] registered [ 0.497665] acpiphp: Slot [26] registered [ 0.499509] acpiphp: Slot [27] registered [ 0.502103] acpiphp: Slot [28] registered [ 0.504181] acpiphp: Slot [29] registered [ 0.507176] acpiphp: Slot [30] registered [ 0.510169] acpiphp: Slot [31] registered [ 0.513074] PCI host bridge to bus 0000:00 [ 0.516022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.521025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.525021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.530023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.532021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.537029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.540322] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.545418] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.552246] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.570576] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.576247] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.580027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.583023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.586018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.589623] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.592872] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.596037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.599825] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.605027] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.632020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.645015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.660080] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.676016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.690018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.732018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.746803] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.754027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.766042] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.784021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.792539] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.797016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.802026] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.818021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.829000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.834000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.834000] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.846017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.858425] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.865015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.871013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.890018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.900615] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.905016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.911016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.924017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.933944] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.935325] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.936397] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.937000] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.937209] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.943287] iommu: Default domain type: Passthrough [ 0.944434] SCSI subsystem initialized [ 0.946128] ACPI: bus type USB registered [ 0.947107] usbcore: registered new interface driver usbfs [ 0.949076] usbcore: registered new interface driver hub [ 0.951089] usbcore: registered new device driver usb [ 0.952156] pps_core: LinuxPPS API ver. 1 registered [ 0.954012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.957061] PTP clock support registered [ 0.960092] EDAC MC: Ver: 3.0.0 [ 0.964119] PCI: Using ACPI for IRQ routing [ 0.965872] NetLabel: Initializing [ 0.967012] NetLabel: domain hash size = 128 [ 0.968009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.969090] NetLabel: unlabeled traffic allowed by default [ 0.971301] vgaarb: loaded [ 0.972312] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.974014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.980473] clocksource: Switched to clocksource kvm-clock [ 1.109619] VFS: Disk quotas dquot_6.6.0 [ 1.111417] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.114243] *** VALIDATE ramfs *** [ 1.115286] *** VALIDATE hugetlbfs *** [ 1.116580] pnp: PnP ACPI init [ 1.119925] pnp: PnP ACPI: found 6 devices [ 1.156655] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.160374] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.162891] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.165451] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.168293] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.170971] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.174345] NET: Registered protocol family 2 [ 1.176951] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.183726] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.187849] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.193732] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.196924] TCP: Hash tables configured (established 65536 bind 65536) [ 1.199518] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.202177] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.204646] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.207356] NET: Registered protocol family 1 [ 1.210427] RPC: Registered named UNIX socket transport module. [ 1.212530] RPC: Registered udp transport module. [ 1.214303] RPC: Registered tcp transport module. [ 1.215844] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.218832] NET: Registered protocol family 44 [ 1.220573] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.222884] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.225469] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.227794] PCI: CLS 0 bytes, default 64 [ 1.229391] Unpacking initramfs... [ 3.294536] debug: unmapping init [mem 0xffff9232bcc54000-0xffff9232bffbffff] [ 3.298821] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.300894] software IO TLB: mapped [mem 0x00000000a3000000-0x00000000a7000000] (64MB) [ 3.304071] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.131730] Initialise system trusted keyrings [ 4.133480] Key type blacklist registered [ 4.135520] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.145189] zbud: loaded [ 4.150145] *** VALIDATE nfs *** [ 4.151410] *** VALIDATE nfs4 *** [ 4.152628] pstore: using deflate compression [ 4.156107] Platform Keyring initialized [ 4.266676] NET: Registered protocol family 38 [ 4.269757] Key type asymmetric registered [ 4.271244] Asymmetric key parser 'x509' registered [ 4.273337] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.276769] io scheduler mq-deadline registered [ 4.278647] io scheduler kyber registered [ 4.280612] io scheduler bfq registered [ 4.282933] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.286416] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.289953] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.293523] ACPI: Power Button [PWRF] [ 4.299911] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.308619] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.320676] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.327060] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.340181] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.375685] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.407091] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.412344] Non-volatile memory driver v1.3 [ 4.413913] Linux agpgart interface v0.103 [ 4.445733] virtio_blk virtio1: [vda] 149768 512-byte logical blocks (76.7 MB/73.1 MiB) [ 4.448732] vda: detected capacity change from 0 to 76681216 [ 4.625759] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.628849] vdb: detected capacity change from 0 to 1073741824 [ 4.658597] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.661624] vdc: detected capacity change from 0 to 2621440000 [ 4.733517] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.737479] vdd: detected capacity change from 0 to 2621440000 [ 4.761637] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.765019] vde: detected capacity change from 0 to 4294967296 [ 4.780453] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.783297] vdf: detected capacity change from 0 to 4294967296 [ 4.789698] libphy: Fixed MDIO Bus: probed [ 4.798364] usbcore: registered new interface driver usbserial_generic [ 4.800374] usbserial: USB Serial support registered for generic [ 4.802473] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.806343] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.808183] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.809992] mousedev: PS/2 mouse device common for all mice [ 4.813402] rtc_cmos 00:05: RTC can wake from S4 [ 4.817454] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.819196] rtc_cmos 00:05: registered as rtc0 [ 4.822334] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.825652] intel_pstate: CPU model not supported [ 4.830398] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.831242] hid: raw HID events driver (C) Jiri Kosina [ 4.838912] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.845454] usbcore: registered new interface driver usbhid [ 4.847844] usbhid: USB HID core driver [ 4.849838] drop_monitor: Initializing network drop monitor service [ 4.852625] Initializing XFRM netlink socket [ 4.855134] NET: Registered protocol family 10 [ 4.858777] Segment Routing with IPv6 [ 4.860552] NET: Registered protocol family 17 [ 4.862941] mpls_gso: MPLS GSO support [ 4.868207] RAS: Correctable Errors collector initialized. [ 4.870606] AVX version of gcm_enc/dec engaged. [ 4.872480] AES CTR mode by8 optimization enabled [ 4.961325] sched_clock: Marking stable (4961286012, 0)->(7026241412, -2064955400) [ 4.966626] registered taskstats version 1 [ 4.968971] Loading compiled-in X.509 certificates [ 4.971295] zswap: loaded using pool lzo/zbud [ 4.996433] Key type big_key registered [ 5.012977] Key type encrypted registered [ 5.014886] ima: No TPM chip found, activating TPM-bypass! [ 5.017471] ima: Allocated hash algorithm: sha1 [ 5.019408] ima: No architecture policies found [ 5.021477] evm: Initialising EVM extended attributes: [ 5.022940] evm: security.selinux [ 5.024600] evm: security.ima [ 5.025832] evm: security.capability [ 5.027218] evm: HMAC attrs: 0x1 [ 5.029694] rtc_cmos 00:05: setting system clock to 2026-09-04 11:52:03 UTC (1788522723) [ 5.036711] debug: unmapping init [mem 0xffffffffbb803000-0xffffffffbb9fffff] [ 5.039800] debug: unmapping init [mem 0xffffffffba582000-0xffffffffba858fff] [ 5.084561] Write protecting the kernel read-only data: 28672k [ 5.109703] debug: unmapping init [mem 0xffffffffb8c03000-0xffffffffb8dfffff] [ 5.112750] debug: unmapping init [mem 0xffffffffb9514000-0xffffffffb95fffff] [ 5.148620] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.156576] systemd[1]: Detected virtualization kvm. [ 5.158491] systemd[1]: Detected architecture x86-64. [ 5.159914] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.418846] systemd[1]: No hostname configured. [ 5.420408] systemd[1]: Set hostname to . [ 5.423798] random: systemd: uninitialized urandom read (16 bytes read) [ 5.425904] systemd[1]: Initializing machine ID from random generator. [ 6.243423] random: systemd: uninitialized urandom read (16 bytes read) [ 6.249966] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 6.259428] random: systemd: uninitialized urandom read (16 bytes read) [ 6.265536] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 6.280522] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ 6.608765] urandom_read: 3 callbacks suppressed [ 6.608780] random: systemd: uninitialized urandom read (16 bytes read) [ OK ] Reached target Swap. [ 6.623861] random: systemd: uninitialized urandom read (16 bytes read) [ OK ] Listening on udev Kernel Socket. [ 6.638621] random: systemd: uninitialized urandom read (16 bytes read) [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.075888] device-mapper: uevent: version 1.0.3 [ 9.086746] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 11.159458] scsi host0: ata_piix [ OK ] Started udev Coldplug all De[ 11.402572] scsi host1: ata_piix vices. [ 11.405859] virtio_net virtio0 ens2: renamed from eth0 [ 11.424860] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.424870] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 11.742681] random: fast init done [ 17.619203] random: crng init done [ 17.620665] random: 1 urandom warning(s) missed due to ratelimiting [ 21.752200] hrtimer: interrupt took 2732363 ns [ 21.817273] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 23.935189] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 28.307996] printk: systemd: 25 output lines suppressed due to ratelimiting [ 29.936758] SELinux: Disabled at runtime. [ 30.124921] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 30.134374] systemd[1]: Detected virtualization kvm. [ 30.136678] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 34.753943] systemd[1]: initrd-switch-root.service: Succeeded. [ 34.795756] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 34.844286] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 34.933686] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 34.964112] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 35.039637] systemd[1]: Starting Journal Service... Starting Journal Service... [ 35.115400] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ 35.258502] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 35.303612] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ 35.762259] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ 35.876123] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 35.924606] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ 35.975841] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ 36.004571] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ 36.137755] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 36.229368] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 36.373636] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ 36.616978] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 41.531158] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 43.790279] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 44.023547] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 46.847366] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 47.107808] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…only root support (14s / no limit) [** ] A start job is running for Configur…only root support (14s / no limit) [*** ] A start job is running for Configur…only root support (15s / no limit) [ *** ] A start job is running for Configur…only root support (15s / no limit) [ *** ] A start job is running for Configur…only root support (16s / no limit) [ ***] A start job is running for Configur…only root support (16s / no limit) [ **] A start job is running for Configur…only root support (17s / no limit) [ *] A start job is running for Configur…only root support (17s / no limit) [ **] A start job is running for Configur…only root support (18s / no limit) [ ***] A start job is running for Configur…only root support (18s / no limit) [ *** ] A start job is running for Configur…only root support (19s / no limit) [ *** ] A start job is running for Configur…only root support (19s / no limit) [*** ] A start job is running for Configur…only root support (20s / no limit) [** ] A start job is running for Configur…only root support (20s / no limit) [* ] A start job is running for Configur…only root support (21s / no limit) [** ] A start job is running for Configur…only root support (21s / no limit) [*** ] A start job is running for Configur…only root support (22s / no limit) [ *** ] A start job is running for Configur…only root support (22s / no limit) [ *** ] A start job is running for Configur…only root support (23s / no limit) [ ***] A start job is running for Configur…only root support (23s / no limit) [ **] A start job is running for Configur…only root support (24s / no limit) [ *] A start job is running for Configur…only root support (24s / no limit) [ **] A start job is running for Configur…only root support (25s / no limit) [ ***] A start job is running for Configur…only root support (25s / no limit)[ 60.215681] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (26s / no limit) [ *** ] A start job is running for Configur…only root support (26s / no limit) [*** ] A start job is running for Configur…only root support (27s / no limit) [** ] A start job is running for Configur…only root support (27s / no limit) [* ] A start job is running for Configur…only root support (28s / no limit) [** ] A start job is running for Configur…only root support (28s / no limit)[ 62.832449] NFS: Registering the id_resolver key type [ 62.855934] Key type id_resolver registered [ 62.860163] Key type id_legacy registered [*** ] A start job is running for Configur…only root support (29s / no limit) [ *** ] A start job is running for Configur…only root support (29s / no limit) [ *** ] A start job is running for Configur…only root support (30s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ ***] A start job is running for Rebuild …amic Linker Cache (37s / no limit) [ **] A start job is running for Rebuild …amic Linker Cache (37s / no limit) [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg145-server login: [ 147.224725] libcfs: loading out-of-tree module taints kernel. [ 147.260390] Key type ._llcrypt registered [ 147.262175] Key type .llcrypt registered [ 147.396398] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 165.228480] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 166.511506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 166.529274] alg: No test for adler32 (adler32-zlib) [ 168.015121] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 168.720089] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 170.431179] Key type lgssc registered [ 172.265632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 189.654568] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 234.037977] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 250.470703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 250.533583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 252.058892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 252.133513] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 252.370948] Lustre: lustre-MDT0000: new disk, initializing [ 252.524194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 252.558240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 257.743508] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 271.055719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 271.229112] Lustre: 6530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 271.282358] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 271.287394] Lustre: Skipped 1 previous similar message [ 271.373993] Lustre: lustre-MDT0001: new disk, initializing [ 271.486440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 271.541253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 271.564244] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 276.887886] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 281.755943] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 290.809287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 291.034519] Lustre: lustre-OST0000: new disk, initializing [ 291.038220] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 291.044137] Lustre: 8465:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 291.104976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 295.048335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 295.062122] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 295.214076] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 297.678531] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 312.905125] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 313.137528] Lustre: lustre-OST0001: new disk, initializing [ 313.145093] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 313.174099] Lustre: 9538:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 313.333454] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 320.657374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 320.671477] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 320.771352] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 321.774948] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 341.016316] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 348.873856] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 357.592653] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing check_logdir /tmp/testlogs/ [ 363.861742] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing yml_node [ 369.113916] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 372.080138] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 375.295408] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 377.234666] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 4 07:58:13 EDT 2026 [ 396.279275] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 397.786322] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 415.200046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 415.208172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 415.231729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 418.282612] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 418.289061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 418.313726] Lustre: Skipped 1 previous similar message [ 418.345469] Lustre: Skipped 3 previous similar messages [ 420.743750] Lustre: server umount lustre-MDT0000 complete [ 424.648257] LustreError: 6520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788523143 with bad export cookie 10931882127757691260 [ 424.656255] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 424.668065] LustreError: 6520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 428.513106] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 428.515730] LustreError: 6534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 428.523491] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 428.536197] Lustre: Skipped 2 previous similar messages [ 428.586119] LustreError: 6534:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 429.536580] LustreError: 9547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 429.557527] LustreError: 9547:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 431.123361] Lustre: server umount lustre-MDT0001 complete [ 435.278704] Lustre: server umount lustre-OST0000 complete [ 439.543523] Lustre: server umount lustre-OST0001 complete [ 446.875469] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 455.929242] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 499.854212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 500.278623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 500.331439] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 500.543450] Lustre: lustre-MDT0000: new disk, initializing [ 500.680982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 500.724641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 506.346966] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 521.040161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 521.250203] Lustre: 15093:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 521.339622] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 521.346691] Lustre: Skipped 1 previous similar message [ 521.462995] Lustre: lustre-MDT0001: new disk, initializing [ 521.604295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 521.689655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 521.717355] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 527.146915] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 538.146650] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 545.454550] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 553.362989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 553.745792] Lustre: lustre-OST0000: new disk, initializing [ 553.762220] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 553.776050] Lustre: 17078:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 553.924846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 555.725394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 555.739668] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 555.874399] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 560.844637] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 573.329325] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 579.310211] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 579.627718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 585.605260] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 586.063688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 591.841938] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 591.844109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 591.856732] LustreError: Skipped 1 previous similar message [ 591.862850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 591.872174] Lustre: Skipped 1 previous similar message [ 591.890804] Lustre: Skipped 4 previous similar messages [ 595.082285] Lustre: server umount lustre-OST0000 complete [ 602.109140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 602.133554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 602.148565] Lustre: Skipped 1 previous similar message [ 612.328843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 612.331885] Lustre: Skipped 2 previous similar messages [ 614.367737] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 614.795395] Lustre: server umount lustre-MDT0000 complete [ 617.442264] LustreError: 18117:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 617.454423] LustreError: 18117:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 619.150897] LustreError: 17089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788523337 with bad export cookie 10931882127757692793 [ 619.153497] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 619.171532] LustreError: 17089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 619.628560] Lustre: server umount lustre-MDT0001 complete [ 625.480423] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:02:21 (1788523341) [ 628.691286] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 631.021627] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:02:26 (1788523346) [ 640.279164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 640.830446] LustreError: 19490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 640.927815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 646.118661] LustreError: 19491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 646.559626] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 656.374260] LustreError: 19491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 656.403112] LustreError: 19491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 656.969834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 663.398729] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 674.436961] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 682.335574] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 690.861185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 699.482610] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 710.398257] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 720.865205] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 720.868914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 720.873163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 720.888879] Lustre: Skipped 2 previous similar messages [ 720.909446] Lustre: Skipped 1 previous similar message [ 725.751179] Lustre: server umount lustre-OST0000 complete [ 737.462536] Lustre: server umount lustre-MDT0000 complete [ 741.348583] LustreError: 19491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 741.363747] LustreError: 19491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 741.986855] LustreError: 19471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788523460 with bad export cookie 10931882127757693675 [ 741.987646] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 741.995185] LustreError: 19471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 742.531779] Lustre: server umount lustre-MDT0001 complete [ 756.885508] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 761.034593] Key type lgssc unregistered [ 761.543683] LNet: 23029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.551747] LNetError: 23029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 761.568784] LNet: Removed LNI 192.168.201.145@tcp [ 763.125865] Key type .llcrypt unregistered [ 763.133452] Key type ._llcrypt unregistered [ 795.137871] Key type ._llcrypt registered [ 795.141168] Key type .llcrypt registered [ 795.386455] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 796.954742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 796.989556] alg: No test for adler32 (adler32-zlib) [ 798.291282] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 798.640505] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 800.375191] Key type lgssc registered [ 801.812460] Lustre: Echo OBD driver; http://www.lustre.org/ [ 813.471951] loop: module loaded [ 813.615486] loop0: detected capacity change from 0 to 204800000 [ 813.947945] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 813.960358] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 814.496748] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 822.669555] loop0: detected capacity change from 0 to 204800000 [ 823.024172] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 823.032491] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 823.049351] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 830.704612] loop0: detected capacity change from 0 to 204800000 [ 839.090857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 839.142591] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 840.582765] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 840.701923] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 840.872410] Lustre: lustre-MDT0000: new disk, initializing [ 840.951642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 841.022528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 847.579339] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 861.270363] loop1: detected capacity change from 0 to 204800000 [ 869.967238] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 870.382682] Lustre: lustre-OST07c6: new disk, initializing [ 870.388764] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 870.394900] Lustre: Skipped 1 previous similar message [ 870.397585] Lustre: 26215:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 870.531101] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 878.865580] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 880.232338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 880.252115] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 880.322562] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 885.747716] Lustre: 25202:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 885.763954] Lustre: 25202:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 885.776118] Lustre: 25202:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 885.786889] Lustre: 25202:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 885.795716] Lustre: 25202:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 885.804765] Lustre: 25202:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 898.345902] Lustre: 25201:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 898.367552] Lustre: 25201:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 898.388422] Lustre: 25201:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 898.396216] Lustre: 25201:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 898.409200] Lustre: 25201:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 898.417489] Lustre: 25201:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 903.946888] Lustre: 25201:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 903.958390] Lustre: 25201:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 903.963075] Lustre: 25201:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 903.966846] Lustre: 25201:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 903.971936] Lustre: 25201:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 903.976212] Lustre: 25201:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 903.984898] Lustre: 25201:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 903.990274] Lustre: 25201:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 903.995253] Lustre: 25201:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 903.999648] Lustre: 25201:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 904.004586] Lustre: 25201:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 904.009254] Lustre: 25201:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 907.316409] Lustre: 25201:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 907.332782] Lustre: 25201:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 907.342984] Lustre: 25201:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 907.353781] Lustre: 25201:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 907.378615] Lustre: 25201:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 907.390452] Lustre: 25201:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 907.406605] Lustre: 25201:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 907.418292] Lustre: 25201:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 907.433655] Lustre: 25201:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 907.438959] Lustre: 25201:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 907.447860] Lustre: 25201:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 907.460533] Lustre: 25201:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 912.664429] Lustre: 25202:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 912.702769] Lustre: 25202:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 912.718218] Lustre: 25202:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 912.730333] Lustre: 25202:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 912.746975] Lustre: 25202:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 912.765992] Lustre: 25202:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 912.778200] Lustre: 25202:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 912.791292] Lustre: 25202:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 912.804530] Lustre: 25202:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 912.812516] Lustre: 25202:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 912.820595] Lustre: 25202:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 912.838719] Lustre: 25202:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 921.930378] Lustre: 25202:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 921.942305] Lustre: 25202:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 921.950262] Lustre: 25202:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 921.967589] Lustre: 25202:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 921.985695] Lustre: 25202:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 922.002218] Lustre: 25202:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 922.018058] Lustre: 25202:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 922.034935] Lustre: 25202:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 922.045101] Lustre: 25202:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 922.059992] Lustre: 25202:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 922.076531] Lustre: 25202:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 922.088828] Lustre: 25202:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 977.377763] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 977.409325] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 977.415420] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 982.498702] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 987.620183] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 990.176120] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 990.553573] Lustre: server umount lustre-OST07c6 complete [ 1001.969952] Lustre: server umount lustre-MDT0000 complete [ 1021.171705] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 1024.161360] Key type lgssc unregistered [ 1024.503189] LNet: 28774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.509724] LNetError: 28774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1025.574907] LNet: Removed LNI 192.168.201.145@tcp [ 1026.642162] Key type .llcrypt unregistered [ 1026.644090] Key type ._llcrypt unregistered [ 1047.199400] Key type ._llcrypt registered [ 1047.203277] Key type .llcrypt registered [ 1047.312210] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 1048.628758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1048.877432] alg: No test for adler32 (adler32-zlib) [ 1049.924857] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 1050.128227] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 1051.823231] Key type lgssc registered [ 1053.118766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1062.757051] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1062.803940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1064.398071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1069.855527] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1073.638588] LustreError: 30041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1073.658077] LustreError: 30041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1078.772336] LustreError: 30042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1079.958178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1080.377757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1085.456786] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1093.058149] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1099.494622] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1106.948858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1107.246881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1109.313794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1114.740945] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1123.655828] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1141.219844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1141.229546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1142.653497] Lustre: server umount lustre-OST0000 complete [ 1146.848682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1146.859300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1146.872769] Lustre: Skipped 1 previous similar message [ 1146.887153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1146.895067] Lustre: Skipped 1 previous similar message [ 1151.469767] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1151.473088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1151.498337] Lustre: Skipped 1 previous similar message [ 1153.009697] Lustre: server umount lustre-MDT0000 complete [ 1156.577066] LustreError: 30041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1156.591865] LustreError: 30041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1156.782309] LustreError: 30023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788523875 with bad export cookie 3321103917257223605 [ 1156.789178] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1156.799630] LustreError: 30023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1157.262507] Lustre: server umount lustre-MDT0001 complete [ 1169.442751] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 1172.501362] Key type lgssc unregistered [ 1172.819361] LNet: 33777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1172.830215] LNetError: 33777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1172.858674] LNet: Removed LNI 192.168.201.145@tcp [ 1173.874893] Key type .llcrypt unregistered [ 1173.884613] Key type ._llcrypt unregistered [ 1195.086127] Key type ._llcrypt registered [ 1195.088210] Key type .llcrypt registered [ 1195.195912] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 1196.488486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1196.590920] alg: No test for adler32 (adler32-zlib) [ 1197.660922] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 1197.975159] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 1199.679284] Key type lgssc registered [ 1201.108960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1209.371907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1209.400520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1210.893409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1215.633974] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1221.091308] LustreError: 35046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1221.117853] LustreError: 35046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1224.613634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1225.317838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1232.028162] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1240.914600] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1247.248233] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1253.753900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1254.031424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1256.113321] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1259.529821] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1267.008715] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1286.121992] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1286.132257] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1286.144129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1288.649541] Lustre: server umount lustre-OST0000 complete [ 1296.356586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1296.361804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1296.370584] Lustre: Skipped 2 previous similar messages [ 1296.377834] Lustre: Skipped 1 previous similar message [ 1299.125237] Lustre: server umount lustre-MDT0000 complete [ 1301.473885] LustreError: 35968:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1301.493594] LustreError: 35968:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1302.677698] LustreError: 35024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788524021 with bad export cookie 10928499642595113419 [ 1302.680958] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1302.690887] LustreError: 35024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1303.134359] Lustre: server umount lustre-MDT0001 complete [ 1313.283365] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 1315.999808] Key type lgssc unregistered [ 1316.295252] LNet: 38602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1316.301965] LNetError: 38602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1317.361662] LNet: Removed LNI 192.168.201.145@tcp [ 1318.330763] Key type .llcrypt unregistered [ 1318.332691] Key type ._llcrypt unregistered [ 1343.403705] Key type ._llcrypt registered [ 1343.405779] Key type .llcrypt registered [ 1343.524793] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 1344.679245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1344.872452] alg: No test for adler32 (adler32-zlib) [ 1345.915895] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 1346.198975] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 1347.863202] Key type lgssc registered [ 1349.382858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1358.390139] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1358.408138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1359.981954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1366.579501] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1369.575607] LustreError: 40012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1369.600325] LustreError: 40012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1374.697281] LustreError: 40013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1379.516185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1379.808966] LustreError: 40012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1380.181490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1385.574402] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1396.987933] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1405.188856] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1413.095697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1413.417264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1420.824441] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1429.825135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1430.223940] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1438.550163] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.45@tcp (stopping) [ 1440.247559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1440.258490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1440.276618] Lustre: Skipped 1 previous similar message [ 1443.648874] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.45@tcp (stopping) [ 1443.664170] Lustre: Skipped 2 previous similar messages [ 1444.183313] Lustre: server umount lustre-MDT0000 complete [ 1445.358952] LustreError: 40013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1445.383262] LustreError: 40013:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1448.625626] LustreError: 39995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788524167 with bad export cookie 4923159640630692036 [ 1448.634538] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1448.638775] LustreError: 39995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1448.791886] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.45@tcp (stopping) [ 1450.467373] LustreError: 40007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1450.468186] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1450.488793] LustreError: 40007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1450.519642] Lustre: Skipped 1 previous similar message [ 1453.474486] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1453.491029] Lustre: Skipped 1 previous similar message [ 1455.125114] Lustre: server umount lustre-MDT0001 complete [ 1477.474771] Lustre: server umount lustre-OST0000 complete [ 1498.175460] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 1501.704773] Key type lgssc unregistered [ 1502.058172] LNet: 43878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1502.071154] LNetError: 43878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1502.086387] LNet: Removed LNI 192.168.201.145@tcp [ 1503.179979] Key type .llcrypt unregistered [ 1503.181732] Key type ._llcrypt unregistered [ 1531.103374] Key type ._llcrypt registered [ 1531.108378] Key type .llcrypt registered [ 1531.205868] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 1532.724827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1532.911721] alg: No test for adler32 (adler32-zlib) [ 1534.028506] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 1534.243314] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 1535.919230] Key type lgssc registered [ 1537.384438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1545.454518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1545.479360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1546.987160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1551.995947] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1557.474924] LustreError: 45291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1557.501859] LustreError: 45291:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1561.914977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1562.359460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1566.675665] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1573.940817] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1580.632467] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1589.564925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1589.999409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1592.058496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1597.086606] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1605.612882] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1617.748760] Lustre: lustre-OST0000: Not available for connect from 192.168.201.45@tcp (stopping) [ 1617.889737] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1617.894153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1617.922259] Lustre: Skipped 1 previous similar message [ 1621.068877] Lustre: server umount lustre-OST0000 complete [ 1639.399349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1639.410344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1639.425650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1639.431254] Lustre: Skipped 2 previous similar messages [ 1643.490365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1643.492200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1645.714548] Lustre: server umount lustre-MDT0000 complete [ 1648.611262] LustreError: 45292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1648.644839] LustreError: 45292:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1650.260574] LustreError: 45273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788524368 with bad export cookie 8006052379221411433 [ 1650.269034] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1650.275619] LustreError: 45273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1656.688773] Lustre: server umount lustre-MDT0001 complete [ 1668.973698] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 1672.125625] Key type lgssc unregistered [ 1672.420741] LNet: 49000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1672.432290] LNetError: 49000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1673.457418] LNet: Removed LNI 192.168.201.145@tcp [ 1674.410963] Key type .llcrypt unregistered [ 1674.414858] Key type ._llcrypt unregistered [ 1700.336807] Key type ._llcrypt registered [ 1700.342230] Key type .llcrypt registered [ 1700.483158] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 1702.062472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1702.186975] alg: No test for adler32 (adler32-zlib) [ 1703.372472] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 1703.661447] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 1705.375210] Key type lgssc registered [ 1706.953540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1714.832542] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1714.850281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1716.416537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1721.581975] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1725.925990] LustreError: 50415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1725.940698] LustreError: 50415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1731.040384] LustreError: 50414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1731.494835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1732.040288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1738.096450] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1746.472506] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1752.143091] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1759.629855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1760.129337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1761.212990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1765.790397] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1773.560352] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1782.151472] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1785.498876] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1797.891742] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-04 8h21m53s [ 1799.712728] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1802.209554] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1802.213869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1802.239677] Lustre: Skipped 1 previous similar message [ 1802.258471] Lustre: Skipped 1 previous similar message [ 1803.318219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1803.330577] Lustre: Skipped 2 previous similar messages [ 1806.147120] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.45@tcp (stopping) [ 1808.296679] Lustre: server umount lustre-MDT0000 complete [ 1809.384604] LustreError: 50415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1809.420519] LustreError: 50415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1812.664470] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1813.293609] LustreError: 50411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1819.443722] LustreError: 52629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1819.461220] LustreError: 52629:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1820.976921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1821.223707] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1821.551810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1821.596846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1822.603179] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1827.118658] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1831.043618] Lustre: DEBUG MARKER: Wait for df (36740) ... [ 1832.662795] Lustre: DEBUG MARKER: done [ 1842.149429] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1842.157517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1842.175585] Lustre: Skipped 1 previous similar message [ 1842.193478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1842.200361] Lustre: Skipped 1 previous similar message [ 1843.189095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1843.201536] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1845.310856] Lustre: server umount lustre-OST0000 complete [ 1853.408850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1853.415600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1853.434735] Lustre: Skipped 1 previous similar message [ 1853.439315] Lustre: Skipped 2 previous similar messages [ 1856.304513] Lustre: server umount lustre-MDT0000 complete [ 1860.581803] LustreError: 50415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1860.613344] LustreError: 50415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1861.137285] LustreError: 50394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788524579 with bad export cookie 15978290189339463357 [ 1861.155194] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1861.155791] LustreError: 50394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1861.743670] Lustre: server umount lustre-MDT0001 complete [ 1873.943595] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 1877.201803] Key type lgssc unregistered [ 1877.477445] LNet: 55339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1877.485984] LNetError: 55339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1877.510825] LNet: Removed LNI 192.168.201.145@tcp [ 1878.310874] Key type .llcrypt unregistered [ 1878.313759] Key type ._llcrypt unregistered [ 1920.021856] Key type ._llcrypt registered [ 1920.028242] Key type .llcrypt registered [ 1920.187466] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 1922.062209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1922.106392] alg: No test for adler32 (adler32-zlib) [ 1923.215322] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 1923.504566] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 1925.207262] Key type lgssc registered [ 1926.583083] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1935.215425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1935.238504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1936.627365] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1936.649535] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1936.866451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1941.741722] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1952.519496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1952.702453] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1952.779519] Lustre: 57546:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1953.182230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1958.508483] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1967.233565] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1974.819557] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1982.520962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1982.673354] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1982.945811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1990.135348] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1990.826283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1999.158369] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2006.365627] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-04 8h25m22s [ 2007.956622] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2016.472636] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2019.396781] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2019.457575] LustreError: 57555:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2039.680149] Lustre: lustre-MDT0000: Client f53923d7-7008-4419-abca-23f9d8c11fc6 (at 192.168.201.45@tcp) reconnecting [ 2059.495179] LustreError: 57555:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2062.721677] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2065.622283] Lustre: DEBUG MARKER: done [ 2077.671672] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2077.688373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2077.708886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2078.176248] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2078.197210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2078.218936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2078.243671] Lustre: Skipped 1 previous similar message [ 2079.533176] Lustre: server umount lustre-OST0000 complete [ 2086.381057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2086.397798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2086.433122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2089.575760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2089.581418] Lustre: Skipped 4 previous similar messages [ 2091.223181] Lustre: server umount lustre-MDT0000 complete [ 2095.775470] LustreError: 57540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788524814 with bad export cookie 14257743778347304933 [ 2095.780682] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2095.800745] LustreError: 57540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2096.282728] Lustre: server umount lustre-MDT0001 complete [ 2110.387978] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 2113.714829] Key type lgssc unregistered [ 2114.115051] LNet: 61877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2114.141256] LNetError: 61877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2114.170423] LNet: Removed LNI 192.168.201.145@tcp [ 2115.179862] Key type .llcrypt unregistered [ 2115.181535] Key type ._llcrypt unregistered [ 2168.779466] Key type ._llcrypt registered [ 2168.783980] Key type .llcrypt registered [ 2168.929840] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 2169.885183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2169.919512] alg: No test for adler32 (adler32-zlib) [ 2171.385156] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 2171.817264] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 2173.591212] Key type lgssc registered [ 2174.832779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2178.848547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2178.860822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2180.127482] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2180.150850] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2180.280977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2181.363192] Lustre: Failing over lustre-MDT0000 [ 2181.818892] Lustre: server umount lustre-MDT0000 complete [ 2190.091178] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 08:28:25 (1788524905) [ 2198.534643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2199.004490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2204.899869] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2214.441924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2214.586335] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2214.649874] Lustre: 64670:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2215.059183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2220.030732] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2229.056361] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2236.765728] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2244.721629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2245.091468] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2245.506230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2250.965120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2254.783420] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2263.337939] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2269.306619] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2276.833311] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2276.865649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2276.875465] Lustre: Skipped 1 previous similar message [ 2281.441933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2281.457220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2281.490710] Lustre: Skipped 1 previous similar message [ 2281.504879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2283.194649] Lustre: server umount lustre-MDT0000 complete [ 2288.270538] LustreError: 64662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788525006 with bad export cookie 15891343783763399426 [ 2288.289994] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2288.294030] LustreError: 64662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2288.806601] Lustre: server umount lustre-MDT0001 complete [ 2293.433822] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2299.451614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2307.554028] Lustre: 63592:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788525010/real 1788525010] req@0000000090f7a1a5 x1875404275099776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788525026 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2307.591944] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2308.576105] Lustre: 63593:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788525011/real 1788525011] req@0000000083fbf06d x1875404275099904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788525027 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2308.943792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2312.673316] Lustre: 63594:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788525015/real 1788525015] req@00000000bcc6bc16 x1875404275100032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788525031 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2313.164582] LustreError: 66519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2313.179801] LustreError: 66519:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2313.271962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2313.436565] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2317.371209] LustreError: 66520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2317.382827] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2318.437329] LustreError: 66519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2318.453275] LustreError: 66519:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2319.145542] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2319.340870] Lustre: 63593:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788525021/real 1788525021] req@000000003b88e0e2 x1875404275100416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788525037 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2319.383825] Lustre: 63593:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2323.943600] LustreError: 68443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2323.965826] LustreError: 68443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2328.738371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2329.058512] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2329.069561] Lustre: Skipped 2 previous similar messages [ 2329.248584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2334.536893] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2334.706327] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2343.627978] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2350.795709] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2359.778094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2359.792185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2359.822340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2359.834727] Lustre: Skipped 1 previous similar message [ 2365.402715] Lustre: server umount lustre-MDT0000 complete [ 2365.408932] LustreError: 68441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2365.435698] LustreError: 68441:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2371.533170] LustreError: 64661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788525090 with bad export cookie 15891343783763401708 [ 2371.542229] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2371.561968] LustreError: 64661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2372.297192] Lustre: server umount lustre-MDT0001 complete [ 2381.634894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2392.033651] Lustre: 63595:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788525094/real 1788525094] req@00000000f8a4f7b1 x1875404275128320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788525110 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2392.081412] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2392.092800] Lustre: Skipped 2 previous similar messages [ 2393.009092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2397.706715] LustreError: 66520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2397.733526] LustreError: 66520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2397.896076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2398.004586] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2400.948164] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2402.942731] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2403.040918] Lustre: 63594:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788525105/real 1788525105] req@0000000000462e78 x1875404275128960/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788525121 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2403.066950] Lustre: 63594:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2415.163113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2421.235515] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2421.279553] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2431.806433] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2439.599675] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2451.938414] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2451.950439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2451.976334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2451.994313] Lustre: Skipped 5 previous similar messages [ 2453.929230] Lustre: server umount lustre-MDT0000 complete [ 2457.067310] LustreError: 72113:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2457.105309] LustreError: 72113:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 2458.828091] LustreError: 64662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788525177 with bad export cookie 15891343783763403087 [ 2458.844091] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2458.848600] LustreError: 64662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2459.259823] Lustre: server umount lustre-MDT0001 complete [ 2467.281674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2471.384676] Lustre: DEBUG MARKER: files compared the same [ 2484.364947] Lustre: server umount lustre-OST0000 complete [ 2505.013176] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 2507.783455] Key type lgssc unregistered [ 2508.083377] LNet: 74403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2508.088704] LNetError: 74403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2508.105405] LNet: Removed LNI 192.168.201.145@tcp [ 2508.873395] Key type .llcrypt unregistered [ 2508.875284] Key type ._llcrypt unregistered [ 2536.998636] Key type ._llcrypt registered [ 2537.003905] Key type .llcrypt registered [ 2537.253430] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 2538.814854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2538.867162] alg: No test for adler32 (adler32-zlib) [ 2540.315971] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 2540.605304] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 2542.337936] Key type lgssc registered [ 2544.191817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2553.694579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2553.740154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2555.419396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2561.017694] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug +malloc all [ 2564.582206] LustreError: 75817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2564.620272] LustreError: 75817:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2569.700483] LustreError: 75818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2571.607033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2572.068309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2578.020645] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug +malloc all [ 2587.518564] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2595.288430] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2603.757326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2604.240580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2612.069044] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug +malloc all [ 2621.393685] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2624.752045] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2640.353249] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2640.355276] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2640.366605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2643.881869] Lustre: server umount lustre-OST0000 complete [ 2649.056164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2649.067033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2649.077054] Lustre: Skipped 1 previous similar message [ 2649.084301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2649.086907] Lustre: Skipped 1 previous similar message [ 2650.598511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2650.601342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2654.795518] Lustre: server umount lustre-MDT0000 complete [ 2655.713201] LustreError: 76547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2655.738091] LustreError: 76547:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2658.952873] LustreError: 75800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788525377 with bad export cookie 15600516818531461644 [ 2658.960551] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2659.286834] Lustre: server umount lustre-MDT0001 complete [ 2671.220826] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 2674.802247] Key type lgssc unregistered [ 2675.209355] LNet: 79372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2675.222281] LNetError: 79372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2675.254605] LNet: Removed LNI 192.168.201.145@tcp [ 2676.330632] Key type .llcrypt unregistered [ 2676.336247] Key type ._llcrypt unregistered [ 2704.402212] Key type ._llcrypt registered [ 2704.405993] Key type .llcrypt registered [ 2704.598514] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 2706.034935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2706.050048] alg: No test for adler32 (adler32-zlib) [ 2707.637845] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 2707.960553] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 2709.650624] Key type lgssc registered [ 2711.389932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2721.079912] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2721.104209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2736.800221] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2741.919349] LustreError: 80759:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.145@tcp: failed processing log, type 4: rc = -110 [ 2768.607293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2775.779974] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2784.516328] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2795.229615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2795.430886] Lustre: *** cfs_fail_loc=706, val=0*** [ 2796.013529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2796.154776] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2801.124820] LustreError: 80785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2801.158441] LustreError: 80785:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2802.374580] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2803.181145] LustreError: 81855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2803.201211] LustreError: 81855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2808.291266] LustreError: 81855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2808.317238] LustreError: 81855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2813.183577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2813.417986] LustreError: 81855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2813.447564] LustreError: 81855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2813.864905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2820.086651] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2830.165328] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2836.639555] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2839.526330] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2839.553248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2839.556302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2844.643766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2844.653539] Lustre: Skipped 1 previous similar message [ 2849.762166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2849.771656] Lustre: Skipped 1 previous similar message [ 2853.343156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2853.585433] Lustre: server umount lustre-OST0000 complete [ 2860.004814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2860.015919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2860.038129] Lustre: Skipped 2 previous similar messages [ 2860.041238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2860.061623] Lustre: Skipped 2 previous similar messages [ 2863.876884] Lustre: server umount lustre-MDT0000 complete [ 2865.120937] LustreError: 81858:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2865.154747] LustreError: 81858:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2867.936843] LustreError: 82595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788525586 with bad export cookie 17355364486730570874 [ 2867.943483] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2867.957301] LustreError: 82595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2868.246501] Lustre: server umount lustre-MDT0001 complete [ 2879.960627] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 2883.091844] Key type lgssc unregistered [ 2883.351400] LNet: 84385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2883.358814] LNetError: 84385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2883.378867] LNet: Removed LNI 192.168.201.145@tcp [ 2884.557550] Key type .llcrypt unregistered [ 2884.559637] Key type ._llcrypt unregistered [ 2911.615312] Key type ._llcrypt registered [ 2911.619255] Key type .llcrypt registered [ 2911.709836] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 2912.865503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2912.966765] alg: No test for adler32 (adler32-zlib) [ 2914.022535] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 2914.283196] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 2915.952642] Key type lgssc registered [ 2917.397882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2926.310789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2926.336509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2931.777962] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2940.036506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2940.412842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2945.524137] LustreError: 86487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2945.550482] LustreError: 86487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2945.708250] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2949.615145] LustreError: 86491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2949.636803] LustreError: 86491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2954.727451] LustreError: 86490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2954.744248] LustreError: 86490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2955.379023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2955.687120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2960.865465] LustreError: 86491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2960.878360] LustreError: 86491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2962.092612] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2964.963777] LustreError: 86490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2972.016475] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2972.042471] Lustre: MGS: Client 979af3e2-f251-41e5-8301-0d4df561f645 (at 0@lo) reconnecting [ 2972.083993] LustreError: 86501:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2972.388686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2972.516770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2977.499822] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2993.126613] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2993.133824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2993.142465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2993.152166] Lustre: Skipped 1 previous similar message [ 2993.163577] Lustre: Skipped 1 previous similar message [ 2997.243459] Lustre: server umount lustre-OST0000 complete [ 3003.361086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3003.361638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3003.399924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3003.411925] Lustre: Skipped 1 previous similar message [ 3008.161830] Lustre: server umount lustre-MDT0000 complete [ 3008.491430] LustreError: 88030:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3008.501031] LustreError: 88030:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3013.669399] Lustre: server umount lustre-MDT0001 complete [ 3018.540993] Lustre: server umount MGS complete [ 3033.746346] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 08:42:28 (1788525748) [ 3066.509444] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 3078.641778] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 3118.998945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3124.623466] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3137.264540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3137.847590] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3137.928746] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3138.114246] Lustre: lustre-MDT0001: new disk, initializing [ 3138.329280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3138.422168] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3138.436971] Lustre: 93660:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3138.466982] LustreError: 93660:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3145.402538] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3158.874814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3159.426392] Lustre: lustre-OST0000: new disk, initializing [ 3159.436914] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3159.454743] Lustre: 94426:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3159.582750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3167.881736] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3170.405057] Lustre: 94851:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000056c6cfdd x1875405053141888/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788525904 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3180.018761] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3188.696510] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3188.709821] Lustre: MGS: Client 2de4bc79-9c94-4b89-a34f-e0c75f1bea42 (at 0@lo) reconnecting [ 3188.762315] Lustre: 92928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3188.776667] LustreError: 93645:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3188.815776] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3188.904967] Lustre: lustre-MDT0000: new disk, initializing [ 3189.060377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3189.112768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3189.133669] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3190.178725] Lustre: 95497:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000534267ab x1875405053149440/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788525924 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3194.173552] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3195.359791] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3197.948262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3197.971148] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3199.283500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 3199.301242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 3199.415541] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 3228.128754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3228.133426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3228.146955] Lustre: Skipped 2 previous similar messages [ 3228.168399] Lustre: Skipped 1 previous similar message [ 3232.494554] Lustre: server umount lustre-OST0000 complete [ 3238.368301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3238.372973] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3238.382199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3238.397293] Lustre: Skipped 1 previous similar message [ 3238.407207] Lustre: Skipped 1 previous similar message [ 3242.882868] Lustre: server umount lustre-MDT0000 complete [ 3243.496125] LustreError: 95749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3243.525964] LustreError: 95749:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3254.247318] Lustre: server umount lustre-MDT0001 complete [ 3258.434674] Lustre: server umount MGS complete [ 3272.132285] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 08:46:28 (1788525988) [ 3294.389617] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 3297.648559] Key type lgssc unregistered [ 3297.999430] LNet: 97993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3298.017418] LNetError: 97993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3298.042216] LNet: Removed LNI 192.168.201.145@tcp [ 3298.869175] Key type .llcrypt unregistered [ 3298.870922] Key type ._llcrypt unregistered [ 3320.547801] Key type ._llcrypt registered [ 3320.555984] Key type .llcrypt registered [ 3320.743503] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 3322.401736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3322.422264] alg: No test for adler32 (adler32-zlib) [ 3323.629775] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 3323.963937] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 3325.679188] Key type lgssc registered [ 3327.286817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3335.960561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3335.987276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3337.446288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3342.622225] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3346.915020] LustreError: 99212:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3346.943702] LustreError: 99212:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3352.034418] LustreError: 99211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3352.271220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3352.535561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3357.394585] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3367.211853] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3374.269930] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3382.620242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3383.203347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3391.637704] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3403.364566] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3406.788175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3417.059625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3417.064255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3417.071706] Lustre: Skipped 1 previous similar message [ 3417.083148] Lustre: Skipped 1 previous similar message [ 3422.178389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3422.204376] Lustre: Skipped 1 previous similar message [ 3423.134753] Lustre: server umount lustre-OST0000 complete [ 3429.350201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3429.365844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3429.400652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3432.416332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3432.417438] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3434.516650] Lustre: server umount lustre-MDT0000 complete [ 3437.542544] LustreError: 101257:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3437.580371] LustreError: 101257:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3438.934187] LustreError: 99191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788526157 with bad export cookie 3431002633309851508 [ 3438.937721] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3438.947186] LustreError: 99191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3439.404482] Lustre: server umount lustre-MDT0001 complete [ 3452.639916] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 3456.260798] Key type lgssc unregistered [ 3456.534895] LNet: 102767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3456.547921] LNetError: 102767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3456.568823] LNet: Removed LNI 192.168.201.145@tcp [ 3457.579518] Key type .llcrypt unregistered [ 3457.581554] Key type ._llcrypt unregistered [ 3477.615627] Key type ._llcrypt registered [ 3477.619321] Key type .llcrypt registered [ 3477.787345] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 3479.205633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3479.370509] alg: No test for adler32 (adler32-zlib) [ 3480.637306] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 3480.973274] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 3482.743256] Key type lgssc registered [ 3484.285539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3492.272973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3492.288174] LustreError: 103842:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3492.422082] LustreError: 103851:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3492.445468] LustreError: 103842:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4866 [ 3492.470245] LustreError: 103842:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3492.479444] LustreError: 103842:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3492.533669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3494.655944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3505.121523] LustreError: 103888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3505.158972] LustreError: 103888:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3506.690789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3507.007976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3513.727165] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3520.831946] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing lsmod [ 3524.952873] LustreError: 105048:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3525.011634] LustreError: 105058:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3525.019190] LustreError: 105048:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4942 [ 3525.033911] LustreError: 105048:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3525.043597] LustreError: 105048:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3525.050329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3525.795956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3529.363536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3529.400123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3532.515400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3532.540148] Lustre: Skipped 4 previous similar messages [ 3536.218439] Lustre: server umount lustre-MDT0000 complete [ 3537.901868] LustreError: 103885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3537.926108] LustreError: 103885:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3541.173681] LustreError: 103868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788526259 with bad export cookie 9705602989204518425 [ 3541.201459] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3542.815388] LustreError: 104319:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002794451a x1875405647664256/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3543.013471] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3543.013715] LustreError: 105243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3543.037307] Lustre: Skipped 2 previous similar messages [ 3543.065280] LustreError: 105243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3543.241507] Lustre: server umount lustre-MDT0001 complete [ 3548.266693] Lustre: server umount lustre-OST0000 complete [ 3558.864029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3559.521909] LustreError: 106120:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3559.618981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3564.538590] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3565.031812] LustreError: 106121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3574.423448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3575.195486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3581.315414] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3591.071680] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3599.209812] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3608.442929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3608.995727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3616.210159] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3625.455555] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3631.558440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3646.949739] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3646.956634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3646.970847] LustreError: Skipped 1 previous similar message [ 3646.973219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3647.000464] Lustre: Skipped 1 previous similar message [ 3647.021807] Lustre: Skipped 1 previous similar message [ 3649.393453] Lustre: server umount lustre-OST0000 complete [ 3657.191422] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3657.200741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3657.210368] Lustre: Skipped 1 previous similar message [ 3657.214861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3657.234225] Lustre: Skipped 1 previous similar message [ 3660.832988] Lustre: server umount lustre-MDT0000 complete [ 3662.308577] LustreError: 106121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3662.344606] LustreError: 106121:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3666.816866] LustreError: 106855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788526385 with bad export cookie 9705602989204519125 [ 3666.817280] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3666.832096] LustreError: 106855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3667.203146] Lustre: server umount lustre-MDT0001 complete [ 3680.478370] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 3683.671850] Key type lgssc unregistered [ 3684.065610] LNet: 109677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3684.074858] LNetError: 109677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3684.093823] LNet: Removed LNI 192.168.201.145@tcp [ 3685.028184] Key type .llcrypt unregistered [ 3685.030220] Key type ._llcrypt unregistered [ 3715.908658] Key type ._llcrypt registered [ 3715.916250] Key type .llcrypt registered [ 3716.070940] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 3717.224930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3717.262546] alg: No test for adler32 (adler32-zlib) [ 3718.438390] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 3718.730109] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 3720.383207] Key type lgssc registered [ 3722.127851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3731.303663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3731.324639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3732.815670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3738.274406] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3743.202626] LustreError: 111091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3743.211699] LustreError: 111091:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3748.324900] LustreError: 111092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3748.684660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3749.125486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3754.534761] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3763.405996] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3770.287646] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3778.790651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3779.220120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3785.510924] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3793.093215] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3802.824630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3809.558783] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3814.112511] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3819.490140] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3819.502140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3819.512068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3823.590397] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3823.596154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3823.602267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3825.170897] Lustre: server umount lustre-OST0000 complete [ 3831.265968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3831.291472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3831.310094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3831.312683] Lustre: Skipped 1 previous similar message [ 3833.825728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3833.829802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3833.844558] Lustre: Skipped 1 previous similar message [ 3837.130206] Lustre: server umount lustre-MDT0000 complete [ 3838.945119] LustreError: 111092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3838.975180] LustreError: 111092:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3842.840120] LustreError: 111073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788526561 with bad export cookie 16280098069234644248 [ 3842.841952] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3842.852099] LustreError: 111073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3843.230666] Lustre: server umount lustre-MDT0001 complete [ 3856.240291] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 3859.954410] Key type lgssc unregistered [ 3860.281766] LNet: 114753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3860.296526] LNetError: 114753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3860.319485] LNet: Removed LNI 192.168.201.145@tcp [ 3861.389746] Key type .llcrypt unregistered [ 3861.391466] Key type ._llcrypt unregistered [ 3883.992982] Key type ._llcrypt registered [ 3883.997954] Key type .llcrypt registered [ 3884.173359] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 3885.802263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3885.882957] alg: No test for adler32 (adler32-zlib) [ 3887.025992] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 3887.343668] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 3889.047244] Key type lgssc registered [ 3890.400246] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3899.842507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3899.859738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3901.503460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3907.307669] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3910.633295] LustreError: 115972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3910.667057] LustreError: 115972:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3915.744510] LustreError: 115973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3917.569779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3918.031678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3923.725464] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3934.375527] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3942.443869] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3951.465880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3951.840968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3958.878363] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3966.991763] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3971.346731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3984.866540] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3984.882539] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3984.907047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3986.913268] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3986.925573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3986.930405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3986.939706] Lustre: Skipped 1 previous similar message [ 3989.407429] Lustre: server umount lustre-OST0000 complete [ 3995.105711] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3995.115045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3995.136592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3997.153858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3997.154023] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4000.816110] Lustre: server umount lustre-MDT0000 complete [ 4002.274660] LustreError: 115972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4002.300079] LustreError: 115972:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4005.805867] LustreError: 115955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788526724 with bad export cookie 10016138139423895080 [ 4005.811439] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4005.820534] LustreError: 115955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4006.457270] Lustre: server umount lustre-MDT0001 complete [ 4019.952091] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 4024.509525] Key type lgssc unregistered [ 4024.909124] LNet: 119531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4024.931964] LNetError: 119531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4024.954883] LNet: Removed LNI 192.168.201.145@tcp [ 4026.173408] Key type .llcrypt unregistered [ 4026.178976] Key type ._llcrypt unregistered [ 4056.530792] Key type ._llcrypt registered [ 4056.532923] Key type .llcrypt registered [ 4056.674519] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 4058.220514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4058.397782] alg: No test for adler32 (adler32-zlib) [ 4059.549224] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 4059.808229] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 4061.528607] Key type lgssc registered [ 4062.932658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4072.520934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4072.534410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4074.337624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4079.894609] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4083.686554] LustreError: 120947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4083.715555] LustreError: 120947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4088.807100] LustreError: 120948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4089.939804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4090.327332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4096.358406] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4105.481469] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4113.792797] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4122.996581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4123.395694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4130.634948] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4139.250847] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4143.863966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 4150.910170] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4155.478305] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4155.484363] Lustre: Skipped 2 previous similar messages [ 4157.125405] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4157.225663] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4160.834789] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4160.846953] Lustre: Skipped 2 previous similar messages [ 4163.352182] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4163.362062] Lustre: Skipped 1 previous similar message [ 4172.553500] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.45@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4172.564853] Lustre: Skipped 2 previous similar messages [ 4173.345379] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.45@tcp 192.168.0.[2,10]@tcp [ 4181.093179] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.45/24@tcp in log lustre-MDT0000 [ 4181.105785] Lustre: Skipped 2 previous similar messages [ 4182.558202] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.45/24@tcp [ 4182.570842] Lustre: Skipped 1 previous similar message [ 4190.180511] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4190.188393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4190.204762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4190.242778] Lustre: Skipped 1 previous similar message [ 4195.301215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4195.317094] Lustre: Skipped 2 previous similar messages [ 4196.166916] Lustre: server umount lustre-OST0000 complete [ 4202.981909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4202.990911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4203.005187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4205.553813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4205.565563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4207.026118] Lustre: server umount lustre-MDT0000 complete [ 4210.667501] LustreError: 120948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4210.703195] LustreError: 120948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4211.515455] LustreError: 120929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788526929 with bad export cookie 922568191117460934 [ 4211.523868] LustreError: 120929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4211.525631] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4211.831692] Lustre: server umount lustre-MDT0001 complete [ 4224.178151] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 4227.448867] Key type lgssc unregistered [ 4227.913676] LNet: 125616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4227.934192] LNetError: 125616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4228.982915] LNet: Removed LNI 192.168.201.145@tcp [ 4230.019518] Key type .llcrypt unregistered [ 4230.023139] Key type ._llcrypt unregistered [ 4271.017688] Key type ._llcrypt registered [ 4271.025503] Key type .llcrypt registered [ 4271.207179] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 4273.157405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4273.212037] alg: No test for adler32 (adler32-zlib) [ 4274.483489] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 4274.825245] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 4276.503190] Key type lgssc registered [ 4277.785893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4284.939084] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4284.975301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4286.432300] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4286.457857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4286.484900] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.45/24@tcp [ 4291.067806] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4295.660843] LustreError: 127174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4295.681916] LustreError: 127174:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4299.721880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4300.021445] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4300.082500] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.45/24@tcp [ 4300.111910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4304.955188] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4312.114625] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4318.776708] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4325.493524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4325.887875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4331.008665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 4332.216739] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4339.631766] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4352.997493] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4353.005997] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4353.021638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4355.552385] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4355.566313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4355.574205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4358.404685] Lustre: server umount lustre-OST0000 complete [ 4365.794578] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4365.795800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4365.812325] Lustre: Skipped 1 previous similar message [ 4365.821399] Lustre: Skipped 2 previous similar messages [ 4368.671311] Lustre: server umount lustre-MDT0000 complete [ 4370.912862] LustreError: 128183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4370.929181] LustreError: 128183:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4372.260805] LustreError: 127156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788527090 with bad export cookie 16843601049928066344 [ 4372.277087] LustreError: 127156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4372.279842] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4372.637401] Lustre: server umount lustre-MDT0001 complete [ 4383.844825] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 4386.761173] Key type lgssc unregistered [ 4387.132744] LNet: 130763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4387.139109] LNetError: 130763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4387.155673] LNet: Removed LNI 192.168.201.145@tcp [ 4388.140302] Key type .llcrypt unregistered [ 4388.144367] Key type ._llcrypt unregistered [ 4415.922263] Key type ._llcrypt registered [ 4415.925303] Key type .llcrypt registered [ 4416.095949] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 4432.422650] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 4433.361111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4433.484479] alg: No test for adler32 (adler32-zlib) [ 4434.609401] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 4434.909115] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 4436.655240] Key type lgssc registered [ 4437.861270] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4473.473133] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4473.505152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4474.775777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4474.802921] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4474.877740] Lustre: lustre-MDT0000: new disk, initializing [ 4474.998963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4475.058299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4480.275702] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4491.747686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4491.898061] Lustre: 134168:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4491.961327] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4491.964021] Lustre: Skipped 1 previous similar message [ 4492.038689] Lustre: lustre-MDT0001: new disk, initializing [ 4492.185778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4492.248036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4492.268852] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4498.006609] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4508.197590] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4513.991530] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4521.866517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4522.245394] Lustre: lustre-OST0000: new disk, initializing [ 4522.250173] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4522.255258] Lustre: 136134:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4522.361137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4528.924802] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4529.248641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4529.270146] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4529.436525] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4538.874810] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4544.918116] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4545.183757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4550.289383] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4550.505153] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4555.233075] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4555.243629] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4555.252240] LustreError: Skipped 1 previous similar message [ 4555.256230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4555.271072] Lustre: Skipped 1 previous similar message [ 4558.739964] Lustre: server umount lustre-OST0000 complete [ 4563.942657] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4563.960975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4563.983891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4563.995437] Lustre: Skipped 1 previous similar message [ 4565.480714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4565.482522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4568.666871] Lustre: server umount lustre-MDT0000 complete [ 4572.444619] LustreError: 134158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788527290 with bad export cookie 486303624258823207 [ 4572.451281] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4572.947734] Lustre: server umount lustre-MDT0001 complete [ 4581.399633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4581.805493] LustreError: 138349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4581.821714] LustreError: 138349:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4581.878401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4586.167950] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4586.978243] LustreError: 138350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4592.097235] LustreError: 138349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4594.936501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4595.375908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4600.144370] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4609.109659] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4615.722086] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4623.922266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4624.158210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4630.298461] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4638.934202] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4643.622984] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4643.833717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4648.371578] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4648.594686] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4657.931508] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4658.233810] Lustre: lustre-OST0001: new disk, initializing [ 4658.239373] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4658.250694] Lustre: 141721:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4658.469379] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4665.472987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4665.478118] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4665.596925] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4667.192156] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4677.563510] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4677.870923] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4684.049451] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4684.344402] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4691.741451] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff94b546a12800.ost_server_uuid 50 [ 4693.657380] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff94b546a12800.ost_server_uuid in IDLE state after 0 sec [ 4706.784334] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4706.787788] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4706.816582] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4706.833593] Lustre: Skipped 2 previous similar messages [ 4708.803258] Lustre: server umount lustre-OST0001 complete [ 4711.913788] LustreError: 141990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4711.925539] LustreError: 141990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4714.469399] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4714.482712] LustreError: Skipped 1 previous similar message [ 4714.489162] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4714.501809] Lustre: Skipped 1 previous similar message [ 4714.506268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4717.031087] LustreError: 141990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4717.042683] LustreError: 141990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4718.863793] Lustre: server umount lustre-OST0000 complete [ 4723.170694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4723.175540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4723.201940] Lustre: Skipped 1 previous similar message [ 4723.211856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4723.214950] Lustre: Skipped 2 previous similar messages [ 4729.360294] Lustre: server umount lustre-MDT0000 complete [ 4732.392047] LustreError: 140792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4732.416930] LustreError: 140792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4733.439113] LustreError: 139084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788527451 with bad export cookie 486303624258824089 [ 4733.457148] LustreError: 139084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4733.463957] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4733.794988] Lustre: server umount lustre-MDT0001 complete [ 4757.296988] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 4760.441410] Key type lgssc unregistered [ 4760.813744] LNet: 144725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4760.819453] LNetError: 144725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4761.832313] LNet: Removed LNI 192.168.201.145@tcp [ 4763.165732] Key type .llcrypt unregistered [ 4763.168055] Key type ._llcrypt unregistered [ 4812.050528] Key type ._llcrypt registered [ 4812.052908] Key type .llcrypt registered [ 4812.139952] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 4829.325118] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 4830.646196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4830.675336] alg: No test for adler32 (adler32-zlib) [ 4831.999445] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 4832.226975] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 4833.916698] Key type lgssc registered [ 4835.352737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4872.438631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4872.463343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4873.682831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4873.711813] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4873.788853] Lustre: lustre-MDT0000: new disk, initializing [ 4873.884911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4873.899444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4878.310756] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4888.548036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4888.699282] Lustre: 148879:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4888.751404] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4888.755777] Lustre: Skipped 1 previous similar message [ 4888.819430] Lustre: lustre-MDT0001: new disk, initializing [ 4888.939480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4888.992225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4889.007259] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4893.495796] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4903.736526] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4910.662756] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4917.045063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4917.403734] Lustre: lustre-OST0000: new disk, initializing [ 4917.416639] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4917.434541] Lustre: 150847:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4917.576444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4924.758759] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4925.129767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4925.146857] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4925.341350] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4934.401603] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4939.436289] Lustre: Failing over lustre-OST0000 [ 4939.713475] Lustre: server umount lustre-OST0000 complete [ 4957.151181] Lustre: 147102:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788527659/real 1788527659] req@00000000917431eb x1875407064302976/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788527675 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4957.155320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4957.185149] Lustre: 147102:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4961.927742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4962.312270] Lustre: 147105:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788527664/real 1788527664] req@000000002c8f1f35 x1875407064304128/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788527680 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4962.341603] Lustre: 147105:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4962.560908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4962.606359] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4966.647660] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4967.647173] Lustre: 147102:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788527669/real 1788527669] req@000000006b413638 x1875407064305408/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788527685 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4967.690644] Lustre: 147102:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4967.806935] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4967.809973] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4971.753624] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4971.999529] Lustre: 147103:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788527674/real 1788527674] req@0000000090dce5dc x1875407064306432/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788527690 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4976.800175] Lustre: Failing over lustre-MDT0000 [ 4977.097661] Lustre: server umount lustre-MDT0000 complete [ 4977.139825] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4977.156870] Lustre: Skipped 3 previous similar messages [ 4982.243982] LustreError: 149947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4982.283163] LustreError: 149947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4983.092479] LustreError: 148886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4987.361347] LustreError: 149945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4987.385339] LustreError: 149945:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4992.489115] LustreError: 148889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4992.520050] LustreError: 148889:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4993.509402] Lustre: 147105:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788527695/real 1788527695] req@0000000017529d94 x1875407064317184/t0(0) o400->MGC192.168.201.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788527711 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4993.557839] Lustre: 147105:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4993.577744] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4998.628059] LustreError: 148886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4998.651719] LustreError: 148886:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4999.239684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5003.777574] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x2b67284c2e66b7b0 [ 5003.789180] Lustre: MGC192.168.201.145@tcp: Connection restored to 0@lo (at 0@lo) [ 5003.797031] Lustre: Skipped 1 previous similar message [ 5004.233835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5004.302421] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5008.688261] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5009.222337] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5009.398771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5009.434834] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5009.505801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5019.107233] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5019.124025] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5019.159112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5024.692616] Lustre: server umount lustre-OST0000 complete [ 5029.857101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5029.857243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5029.862149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5029.862170] Lustre: Skipped 1 previous similar message [ 5029.873430] Lustre: Skipped 2 previous similar messages [ 5034.994310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5035.000307] Lustre: Skipped 2 previous similar messages [ 5035.623413] Lustre: server umount lustre-MDT0000 complete [ 5040.096838] LustreError: 149947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5040.131766] LustreError: 149947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 5040.519303] LustreError: 150846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788527758 with bad export cookie 3127512773893207984 [ 5040.526495] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5040.530552] LustreError: 150846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5041.276784] Lustre: server umount lustre-MDT0001 complete [ 5054.603912] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 5057.960558] Key type lgssc unregistered [ 5058.281615] LNet: 154501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5058.290074] LNetError: 154501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5058.306835] LNet: Removed LNI 192.168.201.145@tcp [ 5059.362805] Key type .llcrypt unregistered [ 5059.365288] Key type ._llcrypt unregistered [ 5099.186818] Key type ._llcrypt registered [ 5099.188901] Key type .llcrypt registered [ 5099.315817] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 5100.658576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5100.679067] alg: No test for adler32 (adler32-zlib) [ 5101.740118] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 5101.987752] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 5103.695179] Key type lgssc registered [ 5105.234043] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5113.155285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5113.175199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5114.696418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5119.950315] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5125.097096] LustreError: 156022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5125.134586] LustreError: 156022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5130.213401] LustreError: 156023:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5131.116701] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5135.332105] LustreError: 156022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5135.646777] LustreError: 156051:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5135.654261] LustreError: 156051:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5135.667917] LustreError: 156051:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 5136.007881] Lustre: server umount lustre-MDT0000 complete [ 5153.197711] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 5165.579548] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 5218.002647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5218.594571] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5218.671177] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5218.961901] Lustre: lustre-MDT0000: new disk, initializing [ 5219.161096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5219.211832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5224.190765] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5238.047091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5238.175153] Lustre: 159359:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5238.258628] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5238.264175] Lustre: Skipped 1 previous similar message [ 5238.323956] Lustre: lustre-MDT0001: new disk, initializing [ 5238.407226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5238.460916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5238.495178] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5242.967883] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5253.395910] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5259.968231] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5267.520746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5267.967996] Lustre: lustre-OST0000: new disk, initializing [ 5267.976182] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5267.984347] Lustre: 161329:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5268.134449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5275.093292] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5276.306497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5276.317955] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5276.501060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5285.497570] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5290.831200] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5292.885486] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 09:20:08 (1788528008) [ 5576.671883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5576.675702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5576.698660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5578.720797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5578.734473] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5578.761659] Lustre: Skipped 1 previous similar message [ 5582.866976] Lustre: server umount lustre-MDT0000 complete [ 5583.843061] LustreError: 159371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5583.865095] LustreError: 159371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5587.769542] LustreError: 159351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788528306 with bad export cookie 5574845813803288648 [ 5587.782840] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5587.801461] LustreError: 159351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5588.959893] LustreError: 163105:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5588.961415] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5589.004851] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5589.009755] Lustre: Skipped 2 previous similar messages [ 5592.991613] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5598.112055] LustreError: 159368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5598.114785] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5598.152693] LustreError: 159368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5602.273621] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5602.691596] Lustre: server umount lustre-MDT0001 complete [ 5622.239160] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5622.603155] Lustre: server umount lustre-OST0000 complete [ 5634.890561] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 5648.400315] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 5693.335809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5693.639855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5693.684760] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5693.794280] Lustre: lustre-MDT0000: new disk, initializing [ 5693.879113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5693.920395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5698.854751] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5712.013176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5712.182717] Lustre: 166010:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5712.233967] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5712.238281] Lustre: Skipped 1 previous similar message [ 5712.327112] Lustre: lustre-MDT0001: new disk, initializing [ 5712.481757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5712.525279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5712.558057] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5718.300295] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5731.108597] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5739.428492] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5749.110471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5749.596882] Lustre: lustre-OST0000: new disk, initializing [ 5749.602294] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5749.621415] Lustre: 167979:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5749.765270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5758.400846] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5759.586164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5759.598173] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5759.673513] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5768.953532] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5777.037689] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5777.402127] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5784.513346] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5784.793646] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5790.694083] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5790.695551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5790.711373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5790.732740] Lustre: Skipped 1 previous similar message [ 5793.559164] Lustre: server umount lustre-OST0000 complete [ 5799.395930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5799.410140] LustreError: Skipped 1 previous similar message [ 5799.416823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5799.432863] Lustre: Skipped 1 previous similar message [ 5803.986716] Lustre: server umount lustre-MDT0000 complete [ 5806.048495] LustreError: 167059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5806.059782] LustreError: 167059:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5807.494981] LustreError: 166002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788528525 with bad export cookie 5574845813803447478 [ 5807.497256] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5807.502614] LustreError: 166002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5807.962762] Lustre: server umount lustre-MDT0001 complete [ 5817.583159] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:28:53 (1788528533) [ 5836.985473] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 5847.964519] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 5889.014708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5889.246159] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5889.271453] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5889.387782] Lustre: lustre-MDT0000: new disk, initializing [ 5889.547621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5889.574605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5895.163641] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5906.641529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5906.796691] Lustre: 172734:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5907.088835] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5912.218445] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5923.216614] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5930.303113] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5938.514449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5938.875040] Lustre: lustre-OST0000: new disk, initializing [ 5938.879691] Lustre: Skipped 1 previous similar message [ 5938.883706] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5938.888880] Lustre: Skipped 2 previous similar messages [ 5938.893694] Lustre: 174713:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5944.449890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5944.470357] Lustre: Skipped 1 previous similar message [ 5944.477397] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5944.681199] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5946.148155] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5956.432190] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5970.399953] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5970.400629] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5970.413241] LustreError: Skipped 1 previous similar message [ 5970.415660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5970.426610] Lustre: Skipped 2 previous similar messages [ 5970.457689] Lustre: Skipped 4 previous similar messages [ 5971.984930] Lustre: server umount lustre-OST0000 complete [ 5978.599352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5982.952824] Lustre: server umount lustre-MDT0000 complete [ 5985.760554] LustreError: 172748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5985.795170] LustreError: 172748:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5987.164578] LustreError: 172729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788528705 with bad export cookie 5574845813803448360 [ 5987.177534] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5987.178162] LustreError: 172729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5987.525159] Lustre: server umount lustre-MDT0001 complete [ 5996.990484] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:31:52 (1788528712) [ 6014.874491] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 6026.170517] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 6068.819083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6069.303312] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6069.357202] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6069.514478] Lustre: lustre-MDT0000: new disk, initializing [ 6069.674891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6069.690910] Lustre: Skipped 2 previous similar messages [ 6069.713071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6075.165088] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6087.982972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6088.170696] Lustre: 179042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6088.499662] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6093.279063] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6104.623136] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6112.086448] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6123.193457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6123.793476] Lustre: 181017:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6131.310444] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6131.578600] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6135.170965] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6149.238221] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6162.407674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6162.429451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6162.430428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6162.430432] Lustre: Skipped 3 previous similar messages [ 6162.479496] Lustre: Skipped 3 previous similar messages [ 6165.344673] Lustre: server umount lustre-OST0000 complete [ 6176.558710] Lustre: server umount lustre-MDT0000 complete [ 6177.762553] LustreError: 179051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6177.776857] LustreError: 179051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6181.486206] LustreError: 179034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788528899 with bad export cookie 5574845813803449445 [ 6181.510145] LustreError: 179034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6181.515703] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6181.891728] Lustre: server umount lustre-MDT0001 complete [ 6195.421791] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 6198.903970] Key type lgssc unregistered [ 6199.184844] LNet: 182875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6199.195662] LNetError: 182875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6199.213172] LNet: Removed LNI 192.168.201.145@tcp [ 6200.047341] Key type .llcrypt unregistered [ 6200.053554] Key type ._llcrypt unregistered [ 6230.132440] Key type ._llcrypt registered [ 6230.138982] Key type .llcrypt registered [ 6230.281443] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 6231.827719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6231.971756] alg: No test for adler32 (adler32-zlib) [ 6233.184832] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 6233.584893] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 6235.303175] Key type lgssc registered [ 6236.974383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6245.438926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6245.475847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6246.935714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6252.471917] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6257.141026] LustreError: 184290:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6257.158954] LustreError: 184290:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6262.245316] LustreError: 184289:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6262.455699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6263.113795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6268.689203] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6278.518976] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6286.303876] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6294.407879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6294.862769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6302.637707] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6312.761993] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6317.434626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6338.025060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6338.052035] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6338.061239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6338.083419] Lustre: Skipped 1 previous similar message [ 6342.594334] Lustre: server umount lustre-OST0000 complete [ 6348.256776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6348.287906] Lustre: Skipped 1 previous similar message [ 6348.297489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6353.194124] Lustre: server umount lustre-MDT0000 complete [ 6353.386342] LustreError: 185243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6353.413480] LustreError: 185243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6358.191498] LustreError: 184271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788529076 with bad export cookie 13753762172450421697 [ 6358.202276] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6358.215799] LustreError: 184271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6358.662843] Lustre: server umount lustre-MDT0001 complete [ 6373.245549] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 6377.677405] Key type lgssc unregistered [ 6378.107797] LNet: 187851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6378.116717] LNetError: 187851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6378.138356] LNet: Removed LNI 192.168.201.145@tcp [ 6379.946520] Key type .llcrypt unregistered [ 6379.951060] Key type ._llcrypt unregistered [ 6413.169402] Key type ._llcrypt registered [ 6413.170753] Key type .llcrypt registered [ 6413.291361] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 6414.677968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6414.759753] alg: No test for adler32 (adler32-zlib) [ 6415.976379] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 6416.346717] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 6418.047178] Key type lgssc registered [ 6419.379536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6428.863799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6428.884376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6430.411455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6435.020866] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6439.909620] LustreError: 189270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6439.946557] LustreError: 189270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6445.042807] LustreError: 189271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6445.589152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6446.308034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6452.302720] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6461.526394] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6469.072838] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6477.265126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6477.779774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6484.801838] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6494.841544] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6500.297176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6505.803701] Lustre: lustre-OST0000: Not available for connect from 192.168.201.45@tcp (stopping) [ 6506.464236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6506.473292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6506.488089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6510.572455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6510.576305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6510.586967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6510.634392] Lustre: Skipped 1 previous similar message [ 6512.142200] Lustre: server umount lustre-OST0000 complete [ 6523.043265] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff94b54e372000.ost_server_uuid 50 [ 6525.237441] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6541.281764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6541.313905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6541.329610] Lustre: Skipped 2 previous similar messages [ 6546.168924] Lustre: server umount lustre-MDT0000 complete [ 6546.400597] LustreError: 189267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6546.423705] LustreError: 189267:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6550.974221] LustreError: 189252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788529269 with bad export cookie 13815686127548289420 [ 6551.005040] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6551.625553] Lustre: server umount lustre-MDT0001 complete [ 6563.370954] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 09:41:18 (1788529278) [ 6571.471802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6571.996705] LustreError: 193188:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6572.110793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6577.125328] LustreError: 193189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6577.156976] LustreError: 193189:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6577.345038] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6587.370557] LustreError: 193188:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6587.401125] LustreError: 193188:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6587.542175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6588.011513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6593.599921] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6603.061915] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6609.953882] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6616.847560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6617.319740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6618.386815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6624.315019] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6632.967356] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6640.950563] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6641.175276] Lustre: lustre-OST0001: new disk, initializing [ 6641.178838] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6641.187678] Lustre: 196030:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6641.360528] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6646.989798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6647.006102] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6647.156576] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6649.815224] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6663.566655] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6667.754816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6667.760518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6667.784255] Lustre: Skipped 2 previous similar messages [ 6667.798657] Lustre: Skipped 1 previous similar message [ 6673.282413] Lustre: server umount lustre-OST0000 complete [ 6674.749223] LustreError: 195010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6682.169578] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6682.411981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6687.812880] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6688.102792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6703.592137] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6703.593749] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6703.599420] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6703.599433] Lustre: Skipped 3 previous similar messages [ 6703.600119] LustreError: Skipped 1 previous similar message [ 6703.617875] Lustre: Skipped 1 previous similar message [ 6708.710874] LustreError: 195010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6708.742030] LustreError: 195010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 6709.299554] Lustre: server umount lustre-OST0001 complete [ 6715.872298] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6715.904305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6720.611761] Lustre: server umount lustre-MDT0000 complete [ 6724.743423] LustreError: 193169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788529443 with bad export cookie 13815686127548290491 [ 6724.750212] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6724.757865] LustreError: 193169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6725.099376] Lustre: server umount lustre-MDT0001 complete [ 6749.809536] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 09:44:25 (1788529465) [ 6757.427660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6757.750911] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6757.784187] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6758.073174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6763.720142] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6775.930237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6775.985961] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6776.014163] Lustre: 199468:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6782.426719] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6795.821448] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6805.339461] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6815.666661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6815.938066] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6823.660137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6824.396584] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6834.991683] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6845.574765] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6845.640412] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6845.845082] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6845.849267] Lustre: Skipped 2 previous similar messages [ 6854.789801] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6864.695933] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6869.984258] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6869.993941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6870.012118] LustreError: Skipped 1 previous similar message [ 6870.038093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6870.043814] Lustre: Skipped 2 previous similar messages [ 6870.053939] Lustre: Skipped 7 previous similar messages [ 6874.617730] Lustre: server umount lustre-OST0000 complete [ 6875.958144] LustreError: 201323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6876.003884] LustreError: 201323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6897.120460] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6897.125835] LustreError: Skipped 1 previous similar message [ 6899.853325] Lustre: server umount lustre-OST0001 complete [ 6904.289006] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6904.305293] Lustre: Skipped 2 previous similar messages [ 6904.321830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6904.324661] Lustre: Skipped 5 previous similar messages [ 6910.779902] Lustre: server umount lustre-MDT0000 complete [ 6915.032689] LustreError: 199462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788529633 with bad export cookie 13815686127548291898 [ 6915.039429] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6915.045486] LustreError: 199462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6915.697675] Lustre: server umount lustre-MDT0001 complete [ 6945.209708] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 09:47:40 (1788529660) [ 6966.517195] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 6979.467691] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 7025.590890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7025.907771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7025.960626] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7026.095244] Lustre: lustre-MDT0000: new disk, initializing [ 7026.242544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7026.272853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7031.966388] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7044.452910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7044.655231] Lustre: 207578:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7044.700219] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7044.703132] Lustre: Skipped 1 previous similar message [ 7044.762826] Lustre: lustre-MDT0001: new disk, initializing [ 7044.896163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7044.917854] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7051.191675] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7061.590274] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7068.292211] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7077.665406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7077.903945] Lustre: lustre-OST0000: new disk, initializing [ 7077.907571] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7077.911656] Lustre: 209550:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7086.754425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7086.775753] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7086.898045] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7087.005815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7100.363742] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7106.062520] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7106.449622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7114.575261] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7114.871529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7117.806975] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7117.815654] LustreError: Skipped 1 previous similar message [ 7117.818963] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7117.840778] Lustre: Skipped 1 previous similar message [ 7117.858837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7117.871496] Lustre: Skipped 3 previous similar messages [ 7123.831939] Lustre: server umount lustre-OST0000 complete [ 7143.397630] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7143.577451] Lustre: server umount lustre-MDT0000 complete [ 7148.234876] LustreError: 208375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788529866 with bad export cookie 13815686127548293459 [ 7148.248932] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7148.759268] Lustre: server umount lustre-MDT0001 complete [ 7158.725276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7159.242506] LustreError: 211767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7159.287909] LustreError: 211767:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 7164.242827] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7174.058895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7180.273555] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7188.651611] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7195.476027] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7205.002280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7216.157883] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7228.507687] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7235.157836] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7235.506729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7240.744095] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7241.070489] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7244.769756] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7244.785160] LustreError: Skipped 2 previous similar messages [ 7249.418606] Lustre: server umount lustre-OST0000 complete [ 7256.388826] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7261.405818] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7294.360136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7294.952780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7294.974108] Lustre: Skipped 5 previous similar messages [ 7302.362696] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7310.309685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7310.331647] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7310.349114] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7311.686972] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7313.875811] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7320.555026] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7320.555795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7320.570121] LustreError: Skipped 1 previous similar message [ 7320.574958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7320.574968] Lustre: Skipped 11 previous similar messages [ 7320.624800] Lustre: Skipped 6 previous similar messages [ 7325.429305] Lustre: server umount lustre-OST0000 complete [ 7336.440595] Lustre: server umount lustre-MDT0000 complete [ 7340.735794] LustreError: 211750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788530059 with bad export cookie 13815686127548294341 [ 7340.748393] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7340.753531] LustreError: 211750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7341.137594] Lustre: server umount lustre-MDT0001 complete [ 7352.564663] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 09:54:28 (1788530068) [ 7360.603476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7367.095120] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7377.491497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7383.199747] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7391.857393] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7399.165868] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7409.192164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7418.559754] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7427.723991] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7432.818988] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7433.138841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7438.237706] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7438.524895] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7445.939905] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7446.139523] Lustre: lustre-OST0001: new disk, initializing [ 7446.141948] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7446.145737] Lustre: 221112:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7451.763794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7451.780182] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7451.920535] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7453.603465] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7464.789510] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7470.208938] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7470.415782] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7475.848587] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7476.197243] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7482.849358] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7482.863399] LustreError: Skipped 1 previous similar message [ 7485.209319] Lustre: server umount lustre-OST0001 complete [ 7487.974471] LustreError: 221385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7488.026969] LustreError: 221385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 7494.973948] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7495.343452] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7503.619426] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7504.197028] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7535.558198] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7541.261452] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7541.278522] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7541.290055] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7541.312760] Lustre: Skipped 1 previous similar message [ 7543.396130] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7553.091560] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7554.904697] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7564.352302] Lustre: server umount lustre-OST0001 complete [ 7585.264055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7585.288858] Lustre: Skipped 6 previous similar messages [ 7585.300981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7585.309555] Lustre: Skipped 9 previous similar messages [ 7589.576949] Lustre: server umount lustre-OST0000 complete [ 7600.854736] Lustre: server umount lustre-MDT0000 complete [ 7604.652277] LustreError: 217721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788530323 with bad export cookie 13815686127548295307 [ 7604.656549] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7604.668988] LustreError: 217721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7611.214988] Lustre: server umount lustre-MDT0001 complete [ 7641.142558] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 09:59:16 (1788530356) [ 7649.193115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7649.569399] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7649.620418] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7654.850766] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7665.244406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7665.362525] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7665.389316] Lustre: 226363:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7670.275841] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7679.813059] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7686.464992] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7695.278732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7695.538217] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7703.060600] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7712.058906] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7722.077158] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7722.187193] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7729.352780] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7738.787535] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7740.542308] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7746.232267] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7746.502914] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7751.507244] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7751.755635] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7758.563796] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff94b54454d000.ost_server_uuid 50 [ 7760.323174] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff94b54454d000.ost_server_uuid in IDLE state after 0 sec [ 7761.349615] Lustre: Permanently deactivating lustre-OST0001 [ 7761.367534] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7766.515829] Lustre: Permanently reactivating lustre-OST0001 [ 7766.535760] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7766.543180] Lustre: Skipped 2 previous similar messages [ 7772.135247] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7772.141589] LustreError: Skipped 4 previous similar messages [ 7777.547192] Lustre: server umount lustre-OST0001 complete [ 7796.194666] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7796.373316] Lustre: server umount lustre-OST0000 complete [ 7806.640875] Lustre: server umount lustre-MDT0000 complete [ 7811.254364] LustreError: 226353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788530529 with bad export cookie 13815686127548296595 [ 7811.266875] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7811.268326] LustreError: 226353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7812.197772] Lustre: server umount lustre-MDT0001 complete [ 7841.687765] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:02:37 (1788530557) [ 7850.218303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7850.446701] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7850.475639] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7850.482141] Lustre: Skipped 2 previous similar messages [ 7850.628218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7850.632958] Lustre: Skipped 9 previous similar messages [ 7855.734852] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7866.634620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7866.744207] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7866.825161] Lustre: 233074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7872.062892] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7880.075175] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7886.588259] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7893.925465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7894.212814] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7894.270486] Lustre: Permanently deactivating lustre-OST0000 [ 7894.291593] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7901.795293] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7903.797838] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7911.432080] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7919.224975] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7919.339635] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7926.545675] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7929.601083] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7934.716103] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7936.761733] Lustre: Permanently reactivating lustre-OST0000 [ 7936.793450] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7936.803710] Lustre: Skipped 2 previous similar messages [ 7938.344032] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7938.358401] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 34 seconds. I think it's dead, and I am evicting it. exp 000000001ba87128, cur 1788530656 deadline 1788530652 last 1788530622 [ 7938.382404] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7938.395323] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7938.499875] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7958.332233] Lustre: server umount lustre-OST0001 complete [ 7968.511123] Lustre: server umount lustre-OST0000 complete [ 7979.477642] Lustre: server umount lustre-MDT0000 complete [ 7983.378937] LustreError: 233064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788530701 with bad export cookie 13815686127548298261 [ 7983.387684] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7983.400550] LustreError: 233064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7984.154641] Lustre: server umount lustre-MDT0001 complete [ 7997.191100] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 8000.209445] Key type lgssc unregistered [ 8000.522765] LNet: 237951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8000.529075] LNetError: 237951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8000.548058] LNet: Removed LNI 192.168.201.145@tcp [ 8001.579367] Key type .llcrypt unregistered [ 8001.581176] Key type ._llcrypt unregistered [ 8027.665498] Key type ._llcrypt registered [ 8027.669426] Key type .llcrypt registered [ 8027.854779] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 8029.682506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8029.733428] alg: No test for adler32 (adler32-zlib) [ 8030.888769] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 8031.215225] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 8032.999187] Key type lgssc registered [ 8034.350775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8045.278780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8045.293065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8046.880681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8052.062979] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 8057.326516] LustreError: 239416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8057.334595] LustreError: 239416:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8062.438557] LustreError: 239415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8062.893184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8063.034163] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8063.093278] Lustre: 239404:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8063.109438] Lustre: Permanently deactivating lustre-MDT0001 [ 8063.124803] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 8063.503377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8068.300769] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 8070.675759] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 8076.445693] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8082.416277] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8088.675937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8089.096315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8092.230763] LustreError: 241258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8092.271292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 8095.762715] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 8097.252848] LustreError: 241258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8102.375432] LustreError: 241259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8103.200356] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8108.534474] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 8110.919046] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8111.169666] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8116.224522] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 8116.251437] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 8118.469653] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 8127.231242] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8130.381432] Lustre: Permanently reactivating lustre-MDT0001 [ 8130.395212] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 8130.398855] Lustre: Skipped 2 previous similar messages [ 8135.010422] Lustre: lustre-MDT0001: Client 790bde6d-53b5-40d8-bc84-86a60bdad02d (at 192.168.201.45@tcp) reconnecting [ 8137.498147] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8137.523545] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 8137.564319] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 8137.602884] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8144.601602] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8146.632396] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8152.966679] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8153.164619] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8158.354105] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8158.657885] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8161.728817] Lustre: Permanently deactivating lustre-MDT0001 [ 8161.742230] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 8161.745420] Lustre: Skipped 2 previous similar messages [ 8170.132401] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 8178.184193] Lustre: server umount lustre-MDT0000 complete [ 8178.663305] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8178.672831] LustreError: 240549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8178.687795] Lustre: Skipped 2 previous similar messages [ 8178.723392] LustreError: 240549:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 8181.630978] LustreError: 239397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788530900 with bad export cookie 12359095590671077277 [ 8181.640084] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8181.643452] LustreError: 239397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8182.222613] Lustre: server umount lustre-MDT0001 complete [ 8186.915900] Lustre: server umount lustre-OST0000 complete [ 8199.887025] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:08:36 (1788530916) [ 8213.690671] Lustre: server umount lustre-OST0001 complete [ 8220.586076] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 8235.362836] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 8275.415552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8275.855278] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8275.870209] Lustre: Skipped 2 previous similar messages [ 8275.895620] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8276.027474] Lustre: lustre-MDT0000: new disk, initializing [ 8276.112293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8276.129708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8281.462713] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 8292.907439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8293.094179] Lustre: 247694:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8293.145806] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8293.152637] Lustre: Skipped 1 previous similar message [ 8293.253692] Lustre: lustre-MDT0001: new disk, initializing [ 8293.442929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8293.486980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8293.512080] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8299.007842] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 8310.094329] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8317.244375] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8325.643269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8325.931835] Lustre: lustre-OST0000: new disk, initializing [ 8325.934923] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8325.943832] Lustre: 249666:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8326.093991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8332.880113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8332.893775] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8332.992738] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 8333.029610] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8344.648324] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8347.249017] LustreError: 247700:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8357.157115] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8357.335137] LustreError: 247700:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8357.360746] LustreError: 249009:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8357.471785] Lustre: lustre-OST0001: new disk, initializing [ 8357.475153] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8357.506348] Lustre: 250781:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8364.150606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8364.162748] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8364.268907] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8364.663573] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 8367.407953] LustreError: 249009:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8375.389899] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8377.036895] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8379.877490] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8379.893058] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8379.914050] Lustre: Skipped 1 previous similar message [ 8379.918927] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8380.752580] Lustre: lustre-OST0001: Not available for connect from 192.168.201.45@tcp (stopping) [ 8380.763509] Lustre: Skipped 1 previous similar message [ 8384.995077] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8385.009310] Lustre: Skipped 1 previous similar message [ 8385.764905] Lustre: server umount lustre-OST0001 complete [ 8385.880589] LustreError: 249649:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8404.960366] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8404.964955] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8404.973960] LustreError: Skipped 2 previous similar messages [ 8404.974650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8405.025571] Lustre: Skipped 2 previous similar messages [ 8410.600206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8410.608540] Lustre: Skipped 4 previous similar messages [ 8418.273058] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8418.744975] Lustre: server umount lustre-OST0000 complete [ 8425.953274] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8425.958332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8425.977584] Lustre: Skipped 1 previous similar message [ 8426.009685] Lustre: Skipped 3 previous similar messages [ 8430.913567] Lustre: server umount lustre-MDT0000 complete [ 8431.072755] LustreError: 247705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8431.096368] LustreError: 247705:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 8435.131670] LustreError: 247687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788531153 with bad export cookie 12359095590671080098 [ 8435.140073] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8435.146533] LustreError: 247687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8441.791881] Lustre: server umount lustre-MDT0001 complete [ 8453.725679] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 8456.795690] Key type lgssc unregistered [ 8457.161565] LNet: 252814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8457.169469] LNetError: 252814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8457.196869] LNet: Removed LNI 192.168.201.145@tcp [ 8458.105986] Key type .llcrypt unregistered [ 8458.110081] Key type ._llcrypt unregistered [ 8503.464385] Key type ._llcrypt registered [ 8503.468656] Key type .llcrypt registered [ 8503.572258] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 8504.766185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8505.000810] alg: No test for adler32 (adler32-zlib) [ 8506.043198] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 8506.307904] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 8507.999193] Key type lgssc registered [ 8509.102971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8516.493263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8516.533136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8517.825595] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8517.858060] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8518.112682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8522.130966] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 8530.413656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8530.479604] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8530.499880] Lustre: 255021:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8530.684105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8534.749483] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 8541.953069] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8547.884616] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8556.178653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8556.314625] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8556.475924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8562.848034] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 8562.933102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8570.877501] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8629.729167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8629.734854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8629.743476] Lustre: Skipped 1 previous similar message [ 8629.758246] Lustre: Skipped 1 previous similar message [ 8634.854138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8634.864811] Lustre: Skipped 1 previous similar message [ 8635.848342] Lustre: server umount lustre-OST0000 complete [ 8640.625706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8653.208559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8653.700295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8659.827756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8659.841987] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8659.842075] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8660.374106] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 8668.686731] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8711.140496] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8711.148093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8711.150832] LustreError: Skipped 1 previous similar message [ 8711.151805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8711.166813] Lustre: Skipped 1 previous similar message [ 8711.190592] Lustre: Skipped 1 previous similar message [ 8714.201433] Lustre: server umount lustre-OST0000 complete [ 8720.352453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8720.369716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8720.389852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8724.089878] Lustre: server umount lustre-MDT0000 complete [ 8727.729359] LustreError: 255014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788531446 with bad export cookie 9679931368219019610 [ 8727.738889] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8727.742056] LustreError: 255014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8728.353895] Lustre: server umount lustre-MDT0001 complete [ 8741.501969] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 8744.152151] Key type lgssc unregistered [ 8744.385631] LNet: 260448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8744.396457] LNetError: 260448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8744.419192] LNet: Removed LNI 192.168.201.145@tcp [ 8745.386249] Key type .llcrypt unregistered [ 8745.390159] Key type ._llcrypt unregistered [ 8771.155185] Key type ._llcrypt registered [ 8771.157278] Key type .llcrypt registered [ 8771.247711] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 8772.408749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8772.437529] alg: No test for adler32 (adler32-zlib) [ 8773.493929] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 8773.726711] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 8775.391176] Key type lgssc registered [ 8776.474487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8784.800736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8784.821246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8786.204234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8790.925889] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 8795.620123] LustreError: 261861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8795.640588] LustreError: 261861:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8799.833685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8800.164977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8805.085603] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 8813.446790] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8819.752402] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8827.620452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8828.072522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8831.162140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8834.057883] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 8843.722808] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8871.394598] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8871.397472] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8871.417432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8876.512792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8876.521443] Lustre: Skipped 2 previous similar messages [ 8877.406553] Lustre: server umount lustre-OST0000 complete [ 8882.145533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8882.158946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8882.188674] Lustre: Skipped 1 previous similar message [ 8882.200049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8886.755819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8886.769562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8888.053465] Lustre: server umount lustre-MDT0000 complete [ 8891.563973] LustreError: 261842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788531610 with bad export cookie 1034916819802709572 [ 8891.569312] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8891.576550] LustreError: 261842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8891.964326] Lustre: server umount lustre-MDT0001 complete [ 8902.436123] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 8905.181920] Key type lgssc unregistered [ 8905.490710] LNet: 266168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8905.495098] LNetError: 266168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8905.507694] LNet: Removed LNI 192.168.201.145@tcp [ 8906.422933] Key type .llcrypt unregistered [ 8906.424802] Key type ._llcrypt unregistered [ 8924.514837] Key type ._llcrypt registered [ 8924.517845] Key type .llcrypt registered [ 8924.618518] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 8925.759810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8925.806266] alg: No test for adler32 (adler32-zlib) [ 8926.780671] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 8927.015439] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 8928.705467] Key type lgssc registered [ 8929.599919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8936.354402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8936.377626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8937.969236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8942.580867] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 8948.194423] LustreError: 267385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8948.213323] LustreError: 267385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8951.547378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8952.003623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8956.719868] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 8963.859542] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8969.108598] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8975.295668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8975.843030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8982.145759] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 8983.034949] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8989.003160] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9003.489335] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9003.502041] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9003.502817] LustreError: Skipped 1 previous similar message [ 9003.503566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9003.527082] Lustre: Skipped 1 previous similar message [ 9006.314338] Lustre: server umount lustre-OST0000 complete [ 9013.216189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9013.219660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9013.236138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9013.243464] Lustre: Skipped 1 previous similar message [ 9016.349895] Lustre: server umount lustre-MDT0000 complete [ 9018.849620] LustreError: 267381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9018.857983] LustreError: 267381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9019.740587] LustreError: 267364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788531738 with bad export cookie 10563709115827944615 [ 9019.743321] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9019.749487] LustreError: 267364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9020.197605] Lustre: server umount lustre-MDT0001 complete [ 9030.173446] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 9032.856572] Key type lgssc unregistered [ 9033.124339] LNet: 271061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9033.132667] LNetError: 271061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9033.151362] LNet: Removed LNI 192.168.201.145@tcp [ 9033.929103] Key type .llcrypt unregistered [ 9033.931075] Key type ._llcrypt unregistered [ 9059.636215] Key type ._llcrypt registered [ 9059.640441] Key type .llcrypt registered [ 9059.735219] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 9060.747582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9060.790195] alg: No test for adler32 (adler32-zlib) [ 9061.939344] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 9062.186570] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 9063.864404] Key type lgssc registered [ 9065.109738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9073.990167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9074.005171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9075.512479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9080.319082] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 9084.899526] LustreError: 272477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9084.926232] LustreError: 272477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9088.936158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9089.227317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9093.602132] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 9101.607241] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9107.820644] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9114.862718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9115.166655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9120.302878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9122.107062] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 9129.449730] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9159.146122] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9159.147824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9159.173233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9164.324797] Lustre: server umount lustre-OST0000 complete [ 9169.381876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9169.387562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9169.397766] Lustre: Skipped 2 previous similar messages [ 9169.403902] Lustre: Skipped 1 previous similar message [ 9174.505905] LustreError: 273206:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9174.560710] Lustre: server umount lustre-MDT0000 complete [ 9177.907599] LustreError: 272459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788531896 with bad export cookie 12533740697577074069 [ 9177.911792] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9177.916809] LustreError: 272459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9178.318040] Lustre: server umount lustre-MDT0001 complete [ 9192.000722] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 9195.106447] Key type lgssc unregistered [ 9195.387404] LNet: 276782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9195.399485] LNetError: 276782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9195.421175] LNet: Removed LNI 192.168.201.145@tcp [ 9196.441526] Key type .llcrypt unregistered [ 9196.445403] Key type ._llcrypt unregistered [ 9216.519255] Key type ._llcrypt registered [ 9216.522302] Key type .llcrypt registered [ 9216.649602] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 9217.444789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9217.523050] alg: No test for adler32 (adler32-zlib) [ 9218.814113] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 9219.090627] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 9220.871242] Key type lgssc registered [ 9221.985396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9230.509391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9230.527680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9231.920462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9235.915812] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 9242.087604] LustreError: 278018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9242.109196] LustreError: 278018:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9243.583028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9243.838117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9247.676794] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 9254.785803] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9261.444506] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9269.152642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9269.456028] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9276.596267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9276.847427] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 9286.158129] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9302.502972] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9302.513636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9302.533146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9304.546614] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9304.550681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9304.563952] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9308.113276] Lustre: server umount lustre-OST0000 complete [ 9314.793226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9314.797939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9314.808020] Lustre: Skipped 1 previous similar message [ 9314.828670] Lustre: Skipped 2 previous similar messages [ 9318.001102] Lustre: server umount lustre-MDT0000 complete [ 9319.906823] LustreError: 278027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9319.932023] LustreError: 278027:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9321.380632] LustreError: 277982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788532039 with bad export cookie 13636157480070833081 [ 9321.386110] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9321.397315] LustreError: 277982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9321.736236] Lustre: server umount lustre-MDT0001 complete [ 9333.024115] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 9335.933439] Key type lgssc unregistered [ 9336.347737] LNet: 281718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9336.351890] LNetError: 281718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9337.381834] LNet: Removed LNI 192.168.201.145@tcp [ 9338.414357] Key type .llcrypt unregistered [ 9338.419113] Key type ._llcrypt unregistered [ 9351.195802] Key type ._llcrypt registered [ 9351.198885] Key type .llcrypt registered [ 9351.283690] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9369.193805] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 9385.162666] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 9386.047402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9386.181110] alg: No test for adler32 (adler32-zlib) [ 9387.165917] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 9387.333712] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 9389.063268] Key type lgssc registered [ 9390.495926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9424.288957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9424.353069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9425.678934] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9425.719845] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9425.889839] Lustre: lustre-MDT0000: new disk, initializing [ 9426.013680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9426.025483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9430.433920] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 9440.631292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9440.736777] Lustre: 285270:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9440.780496] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9440.784316] Lustre: Skipped 1 previous similar message [ 9440.848414] Lustre: lustre-MDT0001: new disk, initializing [ 9440.928179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9440.980202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9440.990336] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9446.116359] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 9456.634923] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9462.939721] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9470.807922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9471.159737] Lustre: lustre-OST0000: new disk, initializing [ 9471.162791] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9471.168045] Lustre: 287236:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9471.265820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9477.186214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9477.202148] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9477.299802] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9477.737510] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 9487.772469] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9493.515463] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9493.740602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9498.966819] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9499.209958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9503.200659] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9503.210513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9503.222617] LustreError: Skipped 1 previous similar message [ 9503.235168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9503.248265] Lustre: Skipped 1 previous similar message [ 9503.275221] Lustre: Skipped 1 previous similar message [ 9508.004160] Lustre: server umount lustre-OST0000 complete [ 9512.939055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9512.946056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9512.966568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9518.561474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9518.568395] Lustre: Skipped 3 previous similar messages [ 9520.726583] Lustre: server umount lustre-MDT0000 complete [ 9525.282478] LustreError: 286063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788532243 with bad export cookie 6955380949970903804 [ 9525.285941] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9525.306441] LustreError: 286063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9526.058287] Lustre: server umount lustre-MDT0001 complete [ 9536.487868] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 10:30:52 (1788532252) [ 9543.440043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9543.840199] LustreError: 289648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9543.852248] LustreError: 289648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9543.897086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9548.236546] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 9549.291049] LustreError: 289649:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9554.408892] LustreError: 289648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9556.680591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9557.185869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9562.227795] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 9570.510736] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9575.908732] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9583.820194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9584.278035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9590.991527] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 9599.678727] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9611.235988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9611.240279] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9611.261235] Lustre: Skipped 1 previous similar message [ 9611.266531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9616.421534] Lustre: server umount lustre-OST0000 complete [ 9622.501200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9622.507306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9622.524622] Lustre: Skipped 2 previous similar messages [ 9622.538130] Lustre: Skipped 3 previous similar messages [ 9626.714392] Lustre: server umount lustre-MDT0000 complete [ 9627.625879] LustreError: 289648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9627.641915] LustreError: 289648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9630.560076] LustreError: 289630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788532349 with bad export cookie 6955380949970904686 [ 9630.568924] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9630.569986] LustreError: 289630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9631.176160] Lustre: server umount lustre-MDT0001 complete [ 9642.371705] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 9645.719986] Key type lgssc unregistered [ 9646.140399] LNet: 293181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9646.157500] LNetError: 293181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9647.212571] LNet: Removed LNI 192.168.201.145@tcp [ 9648.194160] Key type .llcrypt unregistered [ 9648.196407] Key type ._llcrypt unregistered [ 9679.413315] Key type ._llcrypt registered [ 9679.415913] Key type .llcrypt registered [ 9679.512144] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 9694.971978] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 9696.235391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9696.366243] alg: No test for adler32 (adler32-zlib) [ 9697.346746] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 9697.535488] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 9699.175356] Key type lgssc registered [ 9700.583748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9746.529544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9746.559054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9747.967517] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9748.063199] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9748.268761] Lustre: lustre-MDT0000: new disk, initializing [ 9748.395653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9748.427834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9753.617854] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 9764.812377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9764.993649] Lustre: 297006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9765.056249] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9765.059264] Lustre: Skipped 1 previous similar message [ 9765.129202] Lustre: lustre-MDT0001: new disk, initializing [ 9765.229430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9765.285724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9765.300135] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9769.645449] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 9780.839518] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9787.489330] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9794.645794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9794.890794] Lustre: lustre-OST2710: new disk, initializing [ 9794.893940] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9794.899167] Lustre: 298974:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9795.014538] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9801.361870] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 9802.826720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9802.849913] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9802.936924] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9811.090564] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9818.915654] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9819.062403] Lustre: lustre-OST03e8: new disk, initializing [ 9819.065902] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9819.071067] Lustre: 300039:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9819.151882] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9824.158992] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 9825.377109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9825.385709] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9825.458846] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9831.007728] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9837.803704] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9838.027610] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9842.405966] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9842.581763] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9847.622037] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9847.892181] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9851.466923] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9851.647903] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9859.554316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9859.560223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9859.583920] Lustre: Skipped 1 previous similar message [ 9859.602761] Lustre: Skipped 1 previous similar message [ 9864.564515] Lustre: server umount lustre-MDT0000 complete [ 9868.475432] LustreError: 296998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788532586 with bad export cookie 10368508250547456685 [ 9868.478713] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9868.486963] LustreError: 296998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9868.671811] Lustre: server umount lustre-MDT0001 complete [ 9883.372841] Lustre: server umount lustre-OST2710 complete [ 9885.983340] Lustre: 294804:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788532588/real 1788532588] req@000000005cbdd906 x1875412166710144/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788532604 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9886.032317] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9886.039992] Lustre: Skipped 2 previous similar messages [ 9889.251811] Lustre: server umount lustre-OST03e8 complete [ 9910.566628] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 9921.479161] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 9958.212376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9958.461502] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9958.487981] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9958.561227] Lustre: lustre-MDT0000: new disk, initializing [ 9958.638871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9958.651167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9963.434308] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 9974.306575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9974.443477] Lustre: 305289:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9974.513654] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9974.518889] Lustre: Skipped 1 previous similar message [ 9974.621537] Lustre: lustre-MDT0001: new disk, initializing [ 9974.760334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9974.839924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9974.860583] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9979.643361] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 9990.537875] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9997.196898] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10005.135032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10005.496467] Lustre: lustre-OST0000: new disk, initializing [10005.499440] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10005.506456] Lustre: 307258:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10005.607082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10011.572106] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [10013.283081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10013.300306] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10013.405705] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10020.622768] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10025.312079] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10025.534349] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10031.314698] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10031.631609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10039.264326] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10039.270560] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10039.290410] LustreError: Skipped 1 previous similar message [10039.299409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10039.310341] Lustre: Skipped 1 previous similar message [10039.329244] Lustre: Skipped 1 previous similar message [10040.478745] Lustre: server umount lustre-OST0000 complete [10046.434358] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10046.446995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10046.467325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10049.504081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10049.511942] Lustre: Skipped 1 previous similar message [10051.111760] Lustre: server umount lustre-MDT0000 complete [10054.635620] LustreError: 305296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10054.660840] LustreError: 305296:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10055.172567] LustreError: 305282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788532773 with bad export cookie 10368508250547458386 [10055.175792] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10055.180416] LustreError: 305282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10055.682841] Lustre: server umount lustre-MDT0001 complete [10067.327704] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 10:39:42 (1788532782) [10069.213890] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [10071.058973] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 10:39:47 (1788532787) [10075.096522] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [10106.022414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10106.294592] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10106.315897] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10106.525729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10110.900974] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [10119.730312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10119.845001] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10119.873287] Lustre: 311119:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10125.741140] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [10149.544166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10149.856072] Lustre: Denying initial registration attempt from nid 192.168.201.145@tcp, specified as failover [10149.866886] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10149.882295] LustreError: 312570:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [10149.901644] LustreError: 312570:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10149.911558] LustreError: 312570:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10150.014594] Lustre: server umount lustre-OST0000 complete [10150.018088] LustreError: 312570:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [10157.549704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10157.572715] Lustre: Skipped 1 previous similar message [10157.579413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10157.585132] Lustre: Skipped 1 previous similar message [10167.776543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10167.787374] Lustre: Skipped 3 previous similar messages [10171.359400] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10171.507962] Lustre: server umount lustre-MDT0000 complete [10172.896597] LustreError: 311934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10172.906660] LustreError: 311934:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10175.407632] LustreError: 311111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788532893 with bad export cookie 10368508250547459261 [10175.425112] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10175.743809] Lustre: server umount lustre-MDT0001 complete [10185.480708] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [10188.412837] Key type lgssc unregistered [10188.657663] LNet: 313690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10188.664873] LNetError: 313690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10189.737700] LNet: Removed LNI 192.168.201.145@tcp [10190.829482] Key type .llcrypt unregistered [10190.831484] Key type ._llcrypt unregistered [10204.743552] Key type ._llcrypt registered [10204.752868] Key type .llcrypt registered [10204.860589] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [10206.059673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10206.139113] alg: No test for adler32 (adler32-zlib) [10207.244811] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [10207.506653] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [10209.200338] Key type lgssc registered [10210.474457] Lustre: Echo OBD driver; http://www.lustre.org/ [10257.178216] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [10264.384630] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10264.399846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10265.608955] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10265.638968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10265.767698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10269.879971] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [10278.858279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10278.937610] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10278.969188] Lustre: 316453:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10279.183574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10284.217148] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [10291.868377] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10298.938619] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10318.962843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10319.300489] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10319.555751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10325.997810] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [10333.261892] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10335.727116] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10335.736361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10335.754046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10340.322962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10340.337693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10340.359133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10341.749497] Lustre: server umount lustre-OST0000 complete [10345.953506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10345.957047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10345.968428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10345.982403] Lustre: Skipped 2 previous similar messages [10351.073533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10351.081990] Lustre: Skipped 1 previous similar message [10351.539166] Lustre: server umount lustre-MDT0000 complete [10355.680555] LustreError: 317194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788533074 with bad export cookie 14758759397116187766 [10355.684080] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10355.691722] LustreError: 317194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10356.140653] Lustre: server umount lustre-MDT0001 complete [10368.778224] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [10372.050536] Key type lgssc unregistered [10372.446756] LNet: 320025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10372.452914] LNetError: 320025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10372.480330] LNet: Removed LNI 192.168.201.145@tcp [10373.215249] Key type .llcrypt unregistered [10373.219527] Key type ._llcrypt unregistered [10400.331752] Key type ._llcrypt registered [10400.335219] Key type .llcrypt registered [10400.463091] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [10401.603254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10401.740603] alg: No test for adler32 (adler32-zlib) [10402.799575] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [10403.097356] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [10404.855271] Key type lgssc registered [10405.774397] Lustre: Echo OBD driver; http://www.lustre.org/ [10413.830786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10413.852579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10415.557056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10419.725725] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [10424.807825] LustreError: 321439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10424.830629] LustreError: 321439:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10429.566899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10430.121855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10434.208733] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [10442.317672] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10448.991382] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10457.382647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10457.754395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10463.946609] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [10472.370696] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10490.872461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10490.874923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10490.914068] Lustre: Skipped 2 previous similar messages [10490.919929] Lustre: Skipped 2 previous similar messages [10493.661573] Lustre: server umount lustre-MDT0000 complete [10495.975721] LustreError: 321438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10496.021516] LustreError: 321438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10499.046128] LustreError: 321419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788533217 with bad export cookie 16360593040764431556 [10499.054863] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10501.088198] LustreError: 321435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10501.090561] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10501.100442] LustreError: 321435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10501.123117] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10501.128889] Lustre: Skipped 1 previous similar message [10504.096608] LustreError: 321433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10504.103600] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10504.109530] LustreError: 321433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10504.128684] Lustre: Skipped 1 previous similar message [10506.545289] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.45@tcp (stopping) [10509.216833] LustreError: 323690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10509.234989] LustreError: 323690:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10511.660781] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.45@tcp (stopping) [10511.673464] Lustre: Skipped 1 previous similar message [10513.376511] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10513.945775] Lustre: server umount lustre-MDT0001 complete [10522.032140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10533.004920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10544.978639] LustreError: 323270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10545.017756] LustreError: 323270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10545.349697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10545.500925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10551.924825] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [10560.994626] LustreError: 323270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10561.017300] LustreError: 323270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10564.072045] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10564.108712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10564.491572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10569.026385] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [10569.719411] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10578.534691] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10580.734369] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10587.390624] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10589.338961] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10596.337413] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10596.347934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10596.369758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10600.424650] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10600.435313] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10602.027578] Lustre: server umount lustre-OST0000 complete [10610.662704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10610.664378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10610.685257] Lustre: Skipped 1 previous similar message [10612.602820] Lustre: server umount lustre-MDT0000 complete [10615.776166] LustreError: 326262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10615.797469] LustreError: 326262:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10616.958079] LustreError: 324243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788533335 with bad export cookie 16360593040764439284 [10616.960576] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10616.970286] LustreError: 324243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10617.451447] Lustre: server umount lustre-MDT0001 complete [10630.201945] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [10633.472141] Key type lgssc unregistered [10633.765583] LNet: 327729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10633.771504] LNetError: 327729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10633.794736] LNet: Removed LNI 192.168.201.145@tcp [10634.777691] Key type .llcrypt unregistered [10634.783282] Key type ._llcrypt unregistered [10663.900994] Key type ._llcrypt registered [10663.903820] Key type .llcrypt registered [10664.058565] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [10665.018256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10665.115352] alg: No test for adler32 (adler32-zlib) [10666.338627] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [10666.552981] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [10668.240485] Key type lgssc registered [10669.490691] Lustre: Echo OBD driver; http://www.lustre.org/ [10677.633902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10677.685215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10679.150981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10684.292700] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [10689.515715] LustreError: 329145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10689.544693] LustreError: 329145:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10693.736722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10694.213023] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10699.399698] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [10707.066493] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10713.242794] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10719.207659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10719.221943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10719.230446] Lustre: Skipped 1 previous similar message [10719.240492] Lustre: Skipped 1 previous similar message [10723.547490] Lustre: server umount lustre-MDT0000 complete [10724.323694] LustreError: 329873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10724.370797] LustreError: 329873:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10727.676793] LustreError: 329125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788533446 with bad export cookie 1800278283415788389 [10727.686750] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10727.690503] LustreError: 329125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10728.158740] Lustre: server umount lustre-MDT0001 complete [10743.820609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10744.069981] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10744.092227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10744.213527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10748.773864] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [10759.282249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10759.348058] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10759.375242] Lustre: 331731:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10759.735413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10766.194520] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [10776.639322] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10782.966775] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10794.823457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10795.320569] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10795.334485] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10795.341287] LustreError: 333617:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [10795.345350] LustreError: 333617:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10795.360214] LustreError: 333617:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10795.529149] Lustre: server umount lustre-OST0000 complete [10795.537059] LustreError: 333617:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10808.950134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10809.213277] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10809.402120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10815.154926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10817.991829] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [10827.467294] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10836.751979] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10836.838689] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10836.954741] Lustre: lustre-OST0001: new disk, initializing [10836.959762] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10836.969524] Lustre: 335161:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10837.099919] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10844.134717] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [10845.855546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10845.876052] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10846.038285] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10854.033517] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10861.538933] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10861.548067] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10861.574706] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10861.583628] Lustre: Skipped 1 previous similar message [10863.660187] Lustre: server umount lustre-OST0001 complete [10866.667526] LustreError: 334087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10866.690362] LustreError: 334087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10871.777843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10871.784233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10871.785412] LustreError: 334316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10871.792126] LustreError: Skipped 1 previous similar message [10871.795850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10871.843474] Lustre: Skipped 2 previous similar messages [10876.441511] Lustre: server umount lustre-OST0000 complete [10882.016383] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10882.028433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10882.033323] Lustre: Skipped 2 previous similar messages [10887.136587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10887.144437] Lustre: Skipped 1 previous similar message [10896.351142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10896.916159] Lustre: server umount lustre-MDT0000 complete [10897.384563] LustreError: 331744:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10897.400634] LustreError: 331744:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10902.614722] LustreError: 331725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788533621 with bad export cookie 1800278283415788935 [10902.626173] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10902.639878] LustreError: 331725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10903.605567] Lustre: server umount lustre-MDT0001 complete [10919.818748] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [10923.094466] Key type lgssc unregistered [10923.496340] LNet: 337125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10923.502796] LNetError: 337125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10924.522087] LNet: Removed LNI 192.168.201.145@tcp [10925.647928] Key type .llcrypt unregistered [10925.649785] Key type ._llcrypt unregistered [11008.779852] Key type ._llcrypt registered [11008.784023] Key type .llcrypt registered [11008.914751] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [11026.754308] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [11028.418058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11028.472760] alg: No test for adler32 (adler32-zlib) [11029.572369] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [11029.830586] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [11031.559188] Key type lgssc registered [11032.894137] Lustre: Echo OBD driver; http://www.lustre.org/ [11072.031335] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11072.076660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11073.395071] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11073.425126] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11073.515152] Lustre: lustre-MDT0000: new disk, initializing [11073.659500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11073.702635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11079.166804] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [11091.178046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11091.397899] Lustre: 342060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11091.459457] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11091.463393] Lustre: Skipped 1 previous similar message [11091.560681] Lustre: lustre-MDT0001: new disk, initializing [11091.715523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11091.756543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11091.770505] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11097.103529] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [11107.569641] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11113.942689] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11121.481792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11121.955837] Lustre: lustre-OST0000: new disk, initializing [11121.973732] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11121.983875] Lustre: 344029:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11122.172347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11129.741836] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [11132.033538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11132.061777] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11132.232397] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11141.549640] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11147.733591] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11147.978817] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11153.616461] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11153.925512] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11157.985106] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11158.005825] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11158.010162] LustreError: Skipped 1 previous similar message [11158.013784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11158.079573] Lustre: Skipped 1 previous similar message [11163.115321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11163.132052] Lustre: Skipped 2 previous similar messages [11163.342220] Lustre: server umount lustre-OST0000 complete [11168.224832] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11168.228781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11168.243947] Lustre: Skipped 1 previous similar message [11168.261590] Lustre: Skipped 1 previous similar message [11173.346061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11173.356904] Lustre: Skipped 1 previous similar message [11174.541829] Lustre: server umount lustre-MDT0000 complete [11179.690215] LustreError: 342051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788533898 with bad export cookie 9566213128353359947 [11179.699836] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11179.702958] LustreError: 342051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11180.000217] Lustre: server umount lustre-MDT0001 complete [11190.592650] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 10:58:26 (1788533906) [11200.737115] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 10:58:36 (1788533916) [11211.555763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11212.119289] LustreError: 346785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11212.153345] LustreError: 346785:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11212.229683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11217.378348] LustreError: 346786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11218.244171] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [11222.516298] LustreError: 346785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11227.623818] LustreError: 346786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11229.738443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11230.149094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11235.284521] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [11245.057721] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11252.163769] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11260.286913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11260.606234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11267.355458] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [11277.365722] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11290.650395] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11292.588775] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11294.777405] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11296.836928] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11299.292368] Lustre: Failing over lustre-MDT0000 [11299.730937] Lustre: server umount lustre-MDT0000 complete [11301.858836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11301.866561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11301.885437] LustreError: 346785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11311.010419] LustreError: 346780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11311.030606] LustreError: 346780:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [11318.554272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11318.654202] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11318.864609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11318.911910] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11319.095703] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11323.909047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11323.946942] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11324.006378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11324.130320] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [11334.704859] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11336.663955] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11340.557330] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11342.453926] Lustre: Failing over lustre-MDT0000 [11342.687129] Lustre: server umount lustre-MDT0000 complete [11344.360207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11344.367266] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11344.379789] Lustre: Skipped 2 previous similar messages [11344.381185] LustreError: 348335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11344.400936] LustreError: 348335:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [11359.719350] Lustre: 340282:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788534062/real 1788534062] req@0000000030836c2d x1875413563440640/t0(0) o400->MGC192.168.201.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788534078 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11359.755477] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11363.187401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11369.979928] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x84c20428012e32e0 [11370.007240] Lustre: MGC192.168.201.145@tcp: Connection restored to 0@lo (at 0@lo) [11370.020758] Lustre: Skipped 2 previous similar messages [11370.312256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11370.376202] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11371.340618] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11375.340972] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [11375.619684] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11375.653335] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11375.731335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11384.414728] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11386.564258] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11396.066519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11396.073757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11396.083275] Lustre: Skipped 3 previous similar messages [11398.606645] Lustre: server umount lustre-MDT0000 complete [11401.185219] LustreError: 346786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11401.215970] LustreError: 346786:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [11402.230587] LustreError: 347520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788534120 with bad export cookie 9566213128353362656 [11402.234294] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11402.241282] LustreError: 347520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11402.650793] Lustre: server umount lustre-MDT0001 complete [11417.117864] Lustre: server umount lustre-OST0000 complete [11436.003452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11437.217652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11443.315483] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [11455.672780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11462.135370] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [11469.729507] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11475.150187] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11481.183773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11482.627725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11487.928424] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [11495.621957] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11506.145977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11506.161339] Lustre: Skipped 1 previous similar message [11506.176241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11506.179504] Lustre: Skipped 2 previous similar messages [11508.098121] Lustre: server umount lustre-OST0000 complete [11512.800596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11518.424736] Lustre: server umount lustre-MDT0000 complete [11521.511570] LustreError: 355653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11521.539079] LustreError: 355653:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11522.358042] LustreError: 353141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788534240 with bad export cookie 9566213128353363412 [11522.360110] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11522.367638] LustreError: 353141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11522.839691] Lustre: server umount lustre-MDT0001 complete [11535.802911] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [11538.918422] Key type lgssc unregistered [11539.251913] LNet: 356691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11539.257627] LNetError: 356691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11540.326152] LNet: Removed LNI 192.168.201.145@tcp [11541.131626] Key type .llcrypt unregistered [11541.133403] Key type ._llcrypt unregistered [11567.498948] Key type ._llcrypt registered [11567.503297] Key type .llcrypt registered [11567.620795] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [11568.782266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11568.864240] alg: No test for adler32 (adler32-zlib) [11569.884360] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [11570.115119] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [11571.815786] Key type lgssc registered [11573.009743] Lustre: Echo OBD driver; http://www.lustre.org/ [11579.204453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11579.221574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11580.553655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11584.518846] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [11590.112653] LustreError: 358204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11590.131812] LustreError: 358204:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11592.383841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11592.802269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11596.420322] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [11602.562203] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11608.041471] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11614.906754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11615.297889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11621.140895] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [11624.578925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11628.787915] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11638.753497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11638.770328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11638.806906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11640.302149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11640.305277] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11640.319602] Lustre: Skipped 1 previous similar message [11640.601475] Lustre: server umount lustre-MDT0000 complete [11644.162981] LustreError: 358184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788534362 with bad export cookie 11988202871129691918 [11644.171957] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11644.173272] LustreError: 358184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11644.609833] Lustre: server umount lustre-MDT0001 complete [11658.664711] Lustre: server umount lustre-OST0000 complete [11674.288312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11674.987631] LustreError: 361915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11675.130993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11680.237186] LustreError: 361915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11680.259957] LustreError: 361915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11680.350395] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [11685.357066] LustreError: 361916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11688.643779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11689.094860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11693.347681] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [11701.187651] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11707.285675] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11713.867731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11714.015975] LustreError: 363738:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11714.410190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11720.177519] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [11720.699777] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11727.147533] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11740.128477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11740.138687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11740.152236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11740.155131] Lustre: Skipped 2 previous similar messages [11744.364246] Lustre: server umount lustre-MDT0000 complete [11746.279497] LustreError: 361915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11746.301400] LustreError: 361915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11747.580380] LustreError: 364596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788534466 with bad export cookie 11988202871129693031 [11747.581749] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11747.594429] LustreError: 364596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11747.905169] Lustre: server umount lustre-MDT0001 complete [11761.614927] Lustre: server umount lustre-OST0000 complete [11776.088365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11776.520553] LustreError: 365825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11776.602979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11779.963103] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [11787.682175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11791.714447] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [11799.954813] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11805.896423] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11813.551736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11813.958411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11813.965335] Lustre: Skipped 1 previous similar message [11820.476216] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [11822.144879] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11828.518559] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11837.409821] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11837.416636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11837.427433] Lustre: Skipped 2 previous similar messages [11837.433396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11837.437972] Lustre: Skipped 3 previous similar messages [11839.990440] Lustre: server umount lustre-OST0000 complete [11843.553182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11843.557074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11843.570032] Lustre: Skipped 2 previous similar messages [11843.581788] Lustre: Skipped 3 previous similar messages [11850.042462] Lustre: server umount lustre-MDT0000 complete [11853.796191] LustreError: 368383:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11853.813145] LustreError: 368383:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11854.052649] LustreError: 365806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788534572 with bad export cookie 11988202871129694053 [11854.060947] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11854.064984] LustreError: 365806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11854.523640] Lustre: server umount lustre-MDT0001 complete [11866.323565] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [11869.135679] Key type lgssc unregistered [11869.421090] LNet: 369356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11869.432957] LNetError: 369356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11869.446441] LNet: Removed LNI 192.168.201.145@tcp [11870.317704] Key type .llcrypt unregistered [11870.320164] Key type ._llcrypt unregistered [11895.819581] Key type ._llcrypt registered [11895.822572] Key type .llcrypt registered [11895.912294] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [11896.985489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11897.188259] alg: No test for adler32 (adler32-zlib) [11898.220674] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [11898.495513] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [11900.183884] Key type lgssc registered [11901.303662] Lustre: Echo OBD driver; http://www.lustre.org/ [11910.270977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11910.290282] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11910.296260] LustreError: 370793:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11910.329518] LustreError: 370793:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11910.346597] LustreError: 370793:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11910.364828] LustreError: 370793:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11910.384244] LustreError: 370793:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11919.576658] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11919.580833] LustreError: 371193:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11919.605520] LustreError: 371193:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11919.616786] LustreError: 371193:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11919.623459] LustreError: 371193:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11919.628054] LustreError: 371193:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11942.009726] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [11945.610590] Key type lgssc unregistered [11946.026887] LNet: 372194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11946.035572] LNetError: 372194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11946.061754] LNet: Removed LNI 192.168.201.145@tcp [11947.022644] Key type .llcrypt unregistered [11947.026464] Key type ._llcrypt unregistered [11968.961352] Key type ._llcrypt registered [11968.971038] Key type .llcrypt registered [11969.161415] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [11988.308670] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [11989.630167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11989.680181] alg: No test for adler32 (adler32-zlib) [11990.833206] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [11991.139877] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [11992.879312] Key type lgssc registered [11994.239734] Lustre: Echo OBD driver; http://www.lustre.org/ [12033.045843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12033.088773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12034.468208] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12034.499422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12034.578628] Lustre: lustre-MDT0000: new disk, initializing [12034.714796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12034.730341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12039.787467] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [12050.373116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12050.529552] Lustre: 375351:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12050.587884] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12050.595241] Lustre: Skipped 1 previous similar message [12050.674504] Lustre: lustre-MDT0001: new disk, initializing [12050.760736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12050.807464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12050.825687] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12055.677237] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [12064.830191] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12071.125204] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12079.008142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12079.416221] Lustre: lustre-OST0000: new disk, initializing [12079.419281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12079.427400] Lustre: 377317:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12079.530154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12086.276199] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [12086.343810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12086.362802] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12086.516488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12098.129374] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12103.060926] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12103.238859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12108.537802] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12108.806618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12112.370784] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12112.377755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12112.406573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12112.411386] Lustre: Skipped 1 previous similar message [12117.166696] Lustre: server umount lustre-OST0000 complete [12122.593338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12122.599319] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12122.611617] LustreError: Skipped 1 previous similar message [12122.621760] Lustre: Skipped 1 previous similar message [12122.623413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12127.698832] Lustre: server umount lustre-MDT0000 complete [12131.176991] LustreError: 375342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788534849 with bad export cookie 12426749454635554383 [12131.184599] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12131.188769] LustreError: 375342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12131.467388] Lustre: server umount lustre-MDT0001 complete [12139.746389] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12141.639072] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:14:17 (1788534857) [12148.839922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12149.317094] LustreError: 379780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12149.345558] LustreError: 379780:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12149.439795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12153.984655] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [12154.856343] LustreError: 379781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12158.952121] LustreError: 379780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12163.204318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12163.574339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12168.401172] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [12175.939269] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12181.826609] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12188.794252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12189.439618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12196.694918] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [12205.229500] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12212.831071] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12213.000752] Lustre: lustre-OST0001: new disk, initializing [12213.004324] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12213.012828] Lustre: 382616:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12213.117128] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12218.959896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12218.968230] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12219.087224] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12219.797508] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [12232.582586] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12239.702484] Lustre: lustre-OST0001: Not available for connect from 192.168.201.45@tcp (stopping) [12239.708471] Lustre: Skipped 1 previous similar message [12239.843446] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12239.854479] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12239.871324] Lustre: Skipped 2 previous similar messages [12240.160177] Lustre: server umount lustre-OST0001 complete [12244.781836] LustreError: 381603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12250.086768] LustreError: 381601:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12250.115077] LustreError: 381601:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12259.299684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12259.323918] LustreError: Skipped 1 previous similar message [12259.332665] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12259.370512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12259.376975] Lustre: Skipped 2 previous similar messages [12260.325311] LustreError: 381604:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12260.351120] LustreError: 381604:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12265.444262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12265.457377] Lustre: Skipped 3 previous similar messages [12273.120687] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12273.389526] Lustre: server umount lustre-OST0000 complete [12280.803966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12280.817256] Lustre: Skipped 1 previous similar message [12280.820384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12280.823475] Lustre: Skipped 3 previous similar messages [12284.117761] Lustre: server umount lustre-MDT0000 complete [12285.922367] LustreError: 379776:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12285.945367] LustreError: 379776:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12288.670468] LustreError: 379760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788535007 with bad export cookie 12426749454635555265 [12288.676599] LustreError: 379760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12288.682325] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12295.047517] Lustre: server umount lustre-MDT0001 complete [12306.632737] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [12309.343195] Key type lgssc unregistered [12309.583956] LNet: 384605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12309.601175] LNetError: 384605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12309.616898] LNet: Removed LNI 192.168.201.145@tcp [12310.383230] Key type .llcrypt unregistered [12310.385076] Key type ._llcrypt unregistered [12364.983438] Key type ._llcrypt registered [12364.994152] Key type .llcrypt registered [12365.144844] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [12366.415085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12366.560295] alg: No test for adler32 (adler32-zlib) [12367.752692] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [12368.061583] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [12369.711296] Key type lgssc registered [12370.866306] Lustre: Echo OBD driver; http://www.lustre.org/ [12378.564340] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12378.579335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12379.822456] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12379.849737] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12380.004202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12384.864129] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [12395.086701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12395.244798] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12395.287931] Lustre: 387403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12395.678886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12402.033772] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [12413.492475] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12420.526489] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12423.649945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12423.651834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12423.666944] Lustre: Skipped 1 previous similar message [12428.774481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12428.790940] Lustre: Skipped 2 previous similar messages [12429.639258] Lustre: server umount lustre-MDT0000 complete [12433.607820] LustreError: 387394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788535152 with bad export cookie 12535841647158239680 [12433.618793] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12433.627257] LustreError: 387394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12434.307734] Lustre: server umount lustre-MDT0001 complete [12444.759277] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:19:20 (1788535160) [12453.303833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12453.851238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12458.744404] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [12458.978984] LustreError: 389915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12458.996466] LustreError: 389915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12464.109718] LustreError: 389934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12469.128438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12469.221660] LustreError: 389915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12469.472207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12474.598281] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [12483.571624] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12489.788419] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12495.980437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12496.186384] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12496.390894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12503.097373] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [12511.589434] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12520.363798] Lustre: Permanently deactivating lustre-OST0000 [12520.383585] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12529.756659] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12531.390720] LustreError: 392903:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12531.398995] LustreError: 392903:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12531.407455] LustreError: 392903:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12535.946996] LustreError: 392952:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12535.966608] LustreError: 392952:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12540.487241] Lustre: server umount lustre-OST0000 complete [12543.811726] LustreError: 393201:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12543.819207] LustreError: 393201:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12546.547781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12546.559641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12546.607496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12550.115463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12550.122027] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12555.235791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12555.242469] Lustre: Skipped 2 previous similar messages [12556.331917] Lustre: server umount lustre-MDT0000 complete [12560.358326] LustreError: 389916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12560.389831] LustreError: 389916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12560.774543] LustreError: 389897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788535279 with bad export cookie 12535841647158240261 [12560.789703] LustreError: 389897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12560.789759] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12571.327673] Lustre: server umount lustre-MDT0001 complete [12581.519537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12587.353579] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [12590.144411] LustreError: 394360:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12597.116037] Lustre: 394699:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.145@tcp [12599.248535] Lustre: server umount MGS complete [12611.963879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12612.615498] LustreError: 395423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12612.651680] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12612.658352] Lustre: Skipped 1 previous similar message [12612.763434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12618.023925] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [12623.329478] LustreError: 395424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12623.364742] LustreError: 395424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12627.637179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12627.860361] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12633.029907] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [12641.390910] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12648.038934] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12650.144770] Lustre: Permanently reactivating lustre-OST0000 [12650.163523] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12650.168630] Lustre: Skipped 2 previous similar messages [12676.063534] LustreError: 397149:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12684.964554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12685.327868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12685.340700] Lustre: Skipped 1 previous similar message [12691.898363] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [12700.712656] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12711.908834] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12711.925713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12711.928833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12711.959742] Lustre: Skipped 1 previous similar message [12713.462941] Lustre: server umount lustre-OST0000 complete [12720.096868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12720.107644] LustreError: Skipped 1 previous similar message [12720.111719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12732.383276] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12732.388770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12732.415360] Lustre: Skipped 7 previous similar messages [12734.633870] Lustre: server umount lustre-MDT0000 complete [12737.511333] LustreError: 395417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12737.526619] LustreError: 395417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12739.569764] LustreError: 395405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788535458 with bad export cookie 12535841647158241318 [12739.585089] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12739.974382] Lustre: server umount lustre-MDT0001 complete [12752.581842] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [12755.585530] Key type lgssc unregistered [12755.791677] LNet: 399742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12755.796860] LNetError: 399742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12755.809925] LNet: Removed LNI 192.168.201.145@tcp [12756.636816] Key type .llcrypt unregistered [12756.642881] Key type ._llcrypt unregistered [12775.623867] Key type ._llcrypt registered [12775.631804] Key type .llcrypt registered [12775.770622] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [12793.473563] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [12795.173747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12795.230732] alg: No test for adler32 (adler32-zlib) [12796.239858] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [12796.454167] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [12798.103171] Key type lgssc registered [12799.294270] Lustre: Echo OBD driver; http://www.lustre.org/ [12842.922244] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12842.955451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12844.115995] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12844.145894] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12844.208968] Lustre: lustre-MDT0000: new disk, initializing [12844.268326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12844.278479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12847.576766] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [12859.592694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12859.755517] Lustre: 402896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12859.852635] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12859.861567] Lustre: Skipped 1 previous similar message [12859.970270] Lustre: lustre-MDT0001: new disk, initializing [12860.090520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12860.136532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12860.145961] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12864.699709] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [12875.141339] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12881.734784] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12889.494126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12889.816671] Lustre: lustre-OST0000: new disk, initializing [12889.820923] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12889.842419] Lustre: 404864:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12889.925822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12898.406737] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [12900.053080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12900.090948] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12900.352890] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12911.111887] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12916.483043] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12916.716815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12921.946902] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12922.202546] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12925.924538] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12925.952392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12925.965046] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12925.975099] Lustre: Skipped 1 previous similar message [12930.659831] Lustre: server umount lustre-OST0000 complete [12936.165703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12936.194116] Lustre: Skipped 1 previous similar message [12936.198839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12941.280900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12941.288121] Lustre: Skipped 2 previous similar messages [12941.714785] Lustre: server umount lustre-MDT0000 complete [12946.189833] LustreError: 402889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788535664 with bad export cookie 440199289226520404 [12946.194615] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12946.209072] LustreError: 402889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12946.641354] Lustre: server umount lustre-MDT0001 complete [12957.692804] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12959.947272] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:27:55 (1788535675) [12967.543383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12968.309200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12973.552240] LustreError: 407328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12973.588639] LustreError: 407328:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12974.830298] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [12977.636038] LustreError: 407329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12982.758325] LustreError: 407329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12985.053740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12985.548748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12990.618539] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [12998.886316] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13007.051987] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13015.513365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13015.790046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13023.035387] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [13032.826102] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13035.426125] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13043.009342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13050.861308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13050.884888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13050.905661] Lustre: Skipped 1 previous similar message [13050.915768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13053.932162] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13053.935219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13056.585633] Lustre: server umount lustre-OST0000 complete [13062.624984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13062.631404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13062.650547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13062.653698] Lustre: Skipped 2 previous similar messages [13067.394081] Lustre: server umount lustre-MDT0000 complete [13069.284093] LustreError: 407328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13069.306417] LustreError: 407328:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13072.442385] LustreError: 410303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788535790 with bad export cookie 440199289226521286 [13072.453470] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13072.459810] LustreError: 410303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13073.163629] Lustre: server umount lustre-MDT0001 complete [13086.352851] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [13088.999647] Key type lgssc unregistered [13089.345690] LNet: 411054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13089.348614] LNetError: 411054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13089.374304] LNet: Removed LNI 192.168.201.145@tcp [13090.018950] Key type .llcrypt unregistered [13090.021672] Key type ._llcrypt unregistered [13118.682644] Key type ._llcrypt registered [13118.688566] Key type .llcrypt registered [13118.842599] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [13119.301579] Key type .llcrypt unregistered [13119.306397] Key type ._llcrypt unregistered [13141.538316] Key type ._llcrypt registered [13141.544802] Key type .llcrypt registered [13141.670903] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [13142.943232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13142.979615] alg: No test for adler32 (adler32-zlib) [13144.121577] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [13144.463971] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [13146.263207] Key type lgssc registered [13147.604506] Lustre: Echo OBD driver; http://www.lustre.org/ [13156.143514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13156.163226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13157.556409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13161.978732] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [13168.099537] LustreError: 413267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13168.114567] LustreError: 413267:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13171.185693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13171.554401] LustreError: 413268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13171.609854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13176.804173] LustreError: 413267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13176.816479] LustreError: 413267:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13178.426687] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [13181.920738] LustreError: 413268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13181.934173] LustreError: 413268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13182.990223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13188.070781] LustreError: 413267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13188.095729] LustreError: 413267:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13188.313459] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13196.215099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13196.605139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13201.437032] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [13212.118351] Lustre: server umount lustre-OST0000 complete [13217.252286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13217.257589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13217.277405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13217.300269] Lustre: Skipped 1 previous similar message [13222.377477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13222.391795] Lustre: Skipped 1 previous similar message [13227.490882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13227.608172] Lustre: server umount lustre-MDT0000 complete [13228.512256] Lustre: 412576:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788535930/real 1788535930] req@00000000b42d6dd2 x1875415780059904/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788535946 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13228.516689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13228.555039] Lustre: 412576:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [13228.566346] Lustre: Skipped 1 previous similar message [13228.568182] LustreError: 413267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13228.568192] LustreError: 413267:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13231.790207] LustreError: 413248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788535950 with bad export cookie 5900767092341930899 [13231.793660] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13231.802384] LustreError: 413248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13232.300820] Lustre: server umount lustre-MDT0001 complete [13247.206323] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [13250.774991] Key type lgssc unregistered [13251.153878] LNet: 416416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13251.170347] LNetError: 416416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13251.198620] LNet: Removed LNI 192.168.201.145@tcp [13252.432730] Key type .llcrypt unregistered [13252.434708] Key type ._llcrypt unregistered [13288.968817] Key type ._llcrypt registered [13288.973548] Key type .llcrypt registered [13289.064597] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [13290.089592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13290.153451] alg: No test for adler32 (adler32-zlib) [13291.450483] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [13291.726980] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [13293.408993] Key type lgssc registered [13294.837949] Lustre: Echo OBD driver; http://www.lustre.org/ [13303.743054] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13303.763222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13319.403420] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13324.511669] LustreError: 417808:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.145@tcp: failed processing log, type 4: rc = -110 [13351.199890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13358.432794] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [13369.634837] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13378.547128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13379.156606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13379.272527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13384.160872] LustreError: 417833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13384.192075] LustreError: 417833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13384.324940] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [13386.216933] LustreError: 417834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13386.245024] LustreError: 417834:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13391.330499] LustreError: 418846:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13391.371374] LustreError: 418846:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13395.201787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13395.483878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13400.691757] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [13410.315583] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13417.268368] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13425.121554] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13425.131420] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13425.156919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13425.955587] Lustre: server umount lustre-OST0000 complete [13430.762803] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13430.770354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13430.785646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13436.191588] Lustre: server umount lustre-MDT0000 complete [13436.401033] LustreError: 418850:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13436.436272] LustreError: 418850:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13440.351480] LustreError: 417816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788536158 with bad export cookie 13747343680577241917 [13440.367846] LustreError: 417816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13440.367871] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13450.052729] Lustre: server umount lustre-MDT0001 complete [13463.322584] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [13466.707592] Key type lgssc unregistered [13467.179343] LNet: 421379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13467.186839] LNetError: 421379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13468.265135] LNet: Removed LNI 192.168.201.145@tcp [13469.455505] Key type .llcrypt unregistered [13469.462588] Key type ._llcrypt unregistered [13500.652962] Key type ._llcrypt registered [13500.659729] Key type .llcrypt registered [13500.803859] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [13502.173525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13502.201109] alg: No test for adler32 (adler32-zlib) [13503.919768] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [13504.460936] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [13506.289786] Key type lgssc registered [13508.030318] Lustre: Echo OBD driver; http://www.lustre.org/ [13516.983912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13517.011513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13518.539320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13524.396084] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [13528.032795] LustreError: 422799:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13528.057204] LustreError: 422799:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13533.163951] LustreError: 422798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13533.732666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13534.388825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13539.716688] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [13549.011973] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13556.869483] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13565.034288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13565.604385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13573.741222] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [13583.762264] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13588.213230] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13593.570121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13593.584912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13593.589053] Lustre: Skipped 2 previous similar messages [13593.596057] Lustre: Skipped 2 previous similar messages [13595.992280] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.45@tcp (stopping) [13598.694278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13598.700943] Lustre: Skipped 2 previous similar messages [13600.098456] Lustre: server umount lustre-MDT0000 complete [13601.074515] LustreError: 424640:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13603.159858] LustreError: 422777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.45@tcp arrived at 1788536321 with bad export cookie 11286069301320649225 [13603.809925] LustreError: 422799:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13603.843553] LustreError: 422799:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13605.281662] LustreError: 422779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788536323 with bad export cookie 11286069301320648560 [13605.299670] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13605.304746] LustreError: 422779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13608.931633] LustreError: 423526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13608.931903] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13608.947568] LustreError: 423526:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13608.964970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13611.649348] Lustre: server umount lustre-OST0000 complete [13619.090978] Lustre: server umount lustre-MDT0001 complete [13633.936862] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [13637.852757] Key type lgssc unregistered [13638.363242] LNet: 426486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13638.368489] LNetError: 426486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13638.409030] LNet: Removed LNI 192.168.201.145@tcp [13639.802891] Key type .llcrypt unregistered [13639.810081] Key type ._llcrypt unregistered [13670.210955] Key type ._llcrypt registered [13670.217142] Key type .llcrypt registered [13670.362514] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [13671.631585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13671.943685] alg: No test for adler32 (adler32-zlib) [13673.132259] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [13673.469077] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [13675.225174] Key type lgssc registered [13676.601905] Lustre: Echo OBD driver; http://www.lustre.org/ [13684.691653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13684.722652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13686.459391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13691.935417] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [13695.982549] LustreError: 427903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13696.011510] LustreError: 427903:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13701.104350] LustreError: 427902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13701.760675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13702.261110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13706.723572] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [13715.384650] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13722.199962] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13729.604150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13730.107348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13731.131105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13736.894910] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [13745.224308] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13752.114584] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.45@tcp (stopping) [13752.802800] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13752.832175] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13755.935250] Lustre: server umount lustre-MDT0001 complete [13757.236011] LustreError: 427897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13763.058282] LustreError: 427902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13763.098146] LustreError: 427902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13767.136941] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13767.147379] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13767.155514] Lustre: Skipped 1 previous similar message [13767.160275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13767.165251] Lustre: Skipped 1 previous similar message [13768.162136] LustreError: 427903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13770.285085] Lustre: server umount lustre-OST0000 complete [13775.733132] Lustre: server umount lustre-MDT0000 complete [13792.970711] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [13796.099664] Key type lgssc unregistered [13796.395959] LNet: 431586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13796.404366] LNetError: 431586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13796.418656] LNet: Removed LNI 192.168.201.145@tcp [13797.349747] Key type .llcrypt unregistered [13797.352748] Key type ._llcrypt unregistered [13826.186108] Key type ._llcrypt registered [13826.187823] Key type .llcrypt registered [13826.275477] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [13843.040749] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [13844.601439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13844.629671] alg: No test for adler32 (adler32-zlib) [13845.694851] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [13846.027852] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [13847.807186] Key type lgssc registered [13849.096701] Lustre: Echo OBD driver; http://www.lustre.org/ [13897.125706] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [13916.436190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13916.478680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13917.763627] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13917.801421] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13917.885819] Lustre: lustre-MDT0000: new disk, initializing [13917.970277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13918.004770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13923.353247] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [13940.483593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13940.783875] Lustre: lustre-OST0000: new disk, initializing [13940.786362] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13940.789527] Lustre: Skipped 1 previous similar message [13940.793801] Lustre: 436620:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13940.872772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13945.983675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13945.994683] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13946.022023] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13948.545433] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [13959.682764] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13980.213720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13980.428272] Lustre: 435601:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13980.580884] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13980.775072] Lustre: lustre-MDT0001: new disk, initializing [13980.924330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13980.994086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13981.011784] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13986.703575] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [13997.856347] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13998.081673] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [14002.966711] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [14003.240382] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [14009.192871] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.45@tcp (stopping) [14009.316795] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14009.330241] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14014.263307] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.45@tcp (stopping) [14014.278217] Lustre: Skipped 2 previous similar messages [14015.682395] Lustre: server umount lustre-MDT0001 complete [14024.674112] LustreError: 435612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14024.687826] LustreError: 435612:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [14029.818704] LustreError: 435607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14029.854334] LustreError: 435607:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14034.912758] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14034.914657] LustreError: 435613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14034.927723] Lustre: Skipped 1 previous similar message [14034.928752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14034.928757] Lustre: Skipped 2 previous similar messages [14040.034743] LustreError: 435612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14040.036940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14045.180224] LustreError: 435613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14045.184242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14049.247656] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14049.511991] Lustre: server umount lustre-OST0000 complete [14060.229896] Lustre: server umount lustre-MDT0000 complete [14077.731180] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [14081.390734] Key type lgssc unregistered [14081.775179] LNet: 440370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14081.785584] LNetError: 440370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14081.810171] LNet: Removed LNI 192.168.201.145@tcp [14083.040508] Key type .llcrypt unregistered [14083.045664] Key type ._llcrypt unregistered [14132.769844] Key type ._llcrypt registered [14132.773638] Key type .llcrypt registered [14132.940544] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [14133.469448] Key type .llcrypt unregistered [14133.471640] Key type ._llcrypt unregistered [14154.928398] Key type ._llcrypt registered [14154.934633] Key type .llcrypt registered [14155.227783] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [14157.525842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14157.737117] alg: No test for adler32 (adler32-zlib) [14159.379592] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [14159.831932] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [14161.791187] Key type lgssc registered [14163.874601] Lustre: Echo OBD driver; http://www.lustre.org/ [14195.836595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14195.856372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14197.196942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14197.244929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14197.385373] Lustre: lustre-MDT0000: new disk, initializing [14197.551969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14197.586271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14202.454499] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [14214.804352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14214.979335] Lustre: 443750:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14215.073903] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14215.082268] Lustre: Skipped 1 previous similar message [14215.212882] Lustre: lustre-MDT0001: new disk, initializing [14215.385034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14215.447761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14215.463140] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14220.498724] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [14231.689669] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14238.705164] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14245.889451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14246.167589] Lustre: lustre-OST0000: new disk, initializing [14246.173951] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14246.181304] Lustre: 445719:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14246.303788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14253.350482] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [14256.227788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14256.243579] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14256.340239] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14263.441633] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14271.970434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14271.976673] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14271.979950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14272.004204] Lustre: Skipped 2 previous similar messages [14276.953302] Lustre: server umount lustre-MDT0000 complete [14280.906357] LustreError: 443744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788536999 with bad export cookie 16887007368576377949 [14280.910470] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14280.919390] LustreError: 443744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14281.224666] Lustre: server umount lustre-MDT0001 complete [14295.480662] Lustre: server umount lustre-OST0000 complete [14306.511681] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 11:50:22 (1788537022) [14314.815696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14315.572926] LustreError: 447747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14315.603226] LustreError: 447747:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14315.694105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14320.777710] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [14321.130685] LustreError: 447747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14326.241975] LustreError: 447748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14330.510035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14330.945657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14336.232939] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [14345.089945] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14351.578576] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14372.851710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14373.045753] Lustre: Found index 0 for lustre-OST0000, updating log [14373.346882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14380.489829] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [14386.732070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14389.283481] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14396.896464] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14396.906926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14396.911882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14396.911890] Lustre: Skipped 2 previous similar messages [14396.966373] LustreError: Skipped 1 previous similar message [14398.066594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14398.081608] Lustre: Skipped 1 previous similar message [14404.578987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14404.590941] Lustre: Skipped 2 previous similar messages [14409.183422] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14409.561337] Lustre: server umount lustre-OST0000 complete [14428.129958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14428.134979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14428.150583] Lustre: Skipped 1 previous similar message [14428.158914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14431.207395] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14433.062103] Lustre: server umount lustre-MDT0000 complete [14435.359130] LNet: 1 local NIs in recovery (showing 1): 192.168.201.145@tcp [14436.323501] LustreError: 448475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14436.352327] LustreError: 448475:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14437.025964] LustreError: 447727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788537155 with bad export cookie 16887007368576379342 [14437.044444] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14437.640154] Lustre: server umount lustre-MDT0001 complete [14447.276199] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 11:52:43 (1788537163) [15075.130813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15075.750312] LustreError: 454174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15075.944246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15080.933092] LustreError: 454175:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15081.838639] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [15086.049363] LustreError: 454174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15092.194763] LustreError: 454174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15092.218160] LustreError: 454174:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15092.341773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15092.759566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15097.872970] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [15105.962985] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15111.807833] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15126.559460] LNet: 1 local NIs in recovery (showing 1): 192.168.201.145@tcp [15133.159556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15133.414034] Lustre: Found index 0 for lustre-OST0000, updating log [15133.681412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15140.000344] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [15147.922065] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15151.083722] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15151.093616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15151.109762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15151.115874] Lustre: Skipped 2 previous similar messages [15156.534447] Lustre: server umount lustre-OST0000 complete [15161.635423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15161.638859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15161.661507] Lustre: Skipped 1 previous similar message [15161.666973] Lustre: Skipped 1 previous similar message [15163.748658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15163.751310] Lustre: Skipped 2 previous similar messages [15166.885308] LustreError: 454171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15166.900532] LustreError: 454171:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15168.863446] Lustre: server umount lustre-MDT0000 complete [15173.111259] LustreError: 456023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788537891 with bad export cookie 16887007368576380427 [15173.115206] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15173.124378] LustreError: 456023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [15173.436199] Lustre: server umount lustre-MDT0001 complete [15186.976151] LNet: 1 local NIs in recovery (showing 1): 192.168.201.145@tcp [15203.141229] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [15213.803933] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [15252.355191] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:06:08 (1788537968) [15269.731715] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [15272.849282] Key type lgssc unregistered [15273.350833] LNet: 460797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15273.360678] LNetError: 460797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15394.215491] LNet: Removed LNI 192.168.201.145@tcp [15395.287512] Key type .llcrypt unregistered [15395.289416] Key type ._llcrypt unregistered [15413.638330] Key type ._llcrypt registered [15413.640225] Key type .llcrypt registered [15413.732577] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [15414.930024] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15415.088599] alg: No test for adler32 (adler32-zlib) [15416.386759] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [15416.702824] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [15418.343178] Key type lgssc registered [15419.449589] Lustre: Echo OBD driver; http://www.lustre.org/ [15423.784310] LNet: Added LNI 192.168.201.145@tcp100 [8/256/0/180] [15424.555955] LNet: Added LNI 192.168.201.145@tcp101 [8/256/0/180] [15425.278505] LNet: Added LNI 192.168.201.145@tcp102 [8/256/0/180] [15426.227757] LNet: Added LNI 192.168.201.145@tcp103 [8/256/0/180] [15427.144465] LNet: Added LNI 192.168.201.145@tcp104 [8/256/0/180] [15427.926561] LNet: Added LNI 192.168.201.145@tcp105 [8/256/0/180] [15428.766557] LNet: Added LNI 192.168.201.145@tcp106 [8/256/0/180] [15429.707493] LNet: Added LNI 192.168.201.145@tcp107 [8/256/0/180] [15430.790867] LNet: Added LNI 192.168.201.145@tcp108 [8/256/0/180] [15431.922824] LNet: Added LNI 192.168.201.145@tcp109 [8/256/0/180] [15432.711032] LNet: Added LNI 192.168.201.145@tcp110 [8/256/0/180] [15433.685676] LNet: Added LNI 192.168.201.145@tcp111 [8/256/0/180] [15434.629780] LNet: Added LNI 192.168.201.145@tcp112 [8/256/0/180] [15435.484848] LNet: Added LNI 192.168.201.145@tcp113 [8/256/0/180] [15436.407540] LNet: Added LNI 192.168.201.145@tcp114 [8/256/0/180] [15437.339407] LNet: Added LNI 192.168.201.145@tcp115 [8/256/0/180] [15438.278772] LNet: Added LNI 192.168.201.145@tcp116 [8/256/0/180] [15439.296967] LNet: Added LNI 192.168.201.145@tcp117 [8/256/0/180] [15440.160412] LNet: Added LNI 192.168.201.145@tcp118 [8/256/0/180] [15440.957278] LNet: Added LNI 192.168.201.145@tcp119 [8/256/0/180] [15441.894858] LNet: Added LNI 192.168.201.145@tcp120 [8/256/0/180] [15442.872773] LNet: Added LNI 192.168.201.145@tcp121 [8/256/0/180] [15443.738447] LNet: Added LNI 192.168.201.145@tcp122 [8/256/0/180] [15444.663755] LNet: Added LNI 192.168.201.145@tcp123 [8/256/0/180] [15445.595454] LNet: Added LNI 192.168.201.145@tcp124 [8/256/0/180] [15446.476496] LNet: Added LNI 192.168.201.145@tcp125 [8/256/0/180] [15447.242386] LNet: Added LNI 192.168.201.145@tcp126 [8/256/0/180] [15448.208937] LNet: Added LNI 192.168.201.145@tcp127 [8/256/0/180] [15449.097963] LNet: Added LNI 192.168.201.145@tcp128 [8/256/0/180] [15449.846293] LNet: Added LNI 192.168.201.145@tcp129 [8/256/0/180] [15450.767512] LNet: Added LNI 192.168.201.145@tcp130 [8/256/0/180] [15451.712635] LNet: Added LNI 192.168.201.145@tcp131 [8/256/0/180] [15452.681500] LNet: Added LNI 192.168.201.145@tcp132 [8/256/0/180] [15453.450852] LNet: Added LNI 192.168.201.145@tcp133 [8/256/0/180] [15454.433461] LNet: Added LNI 192.168.201.145@tcp134 [8/256/0/180] [15455.394822] LNet: Added LNI 192.168.201.145@tcp135 [8/256/0/180] [15462.532199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15462.549573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15463.647127] Lustre: 463829:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.145@tcp: can't add NID 192.168.201.145@tcp130: rc = -75 [15463.749836] LNetError: 463856:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.145 [15463.767032] LNetError: Protocol error connecting to 192.168.201.145@tcp123 at host 192.168.201.145:988: is it running a compatible version of Lustre? [15463.778066] LNetError: Skipped 5 previous similar messages [15463.840195] LNet: 4 local NIs in recovery (showing 4): 192.168.201.145@tcp126, 192.168.201.145@tcp129, 192.168.201.145@tcp128, 192.168.201.145@tcp123 [15463.881436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15463.929435] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15464.049462] Lustre: lustre-MDT0000: new disk, initializing [15464.154026] Lustre: 463884:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.145@tcp: can't add NID 192.168.201.145@tcp130: rc = -75 [15464.162628] Lustre: 463884:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15464.189682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15464.206467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15464.803346] LNetError: 461287:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.145 [15464.825496] LNetError: 461287:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [15464.838740] LNetError: Protocol error connecting to 192.168.201.145@tcp101 at host 192.168.201.145:988: is it running a compatible version of Lustre? [15466.920497] LNetError: 461288:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.145 [15466.954627] LNetError: 461288:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [15466.967376] LNetError: Protocol error connecting to 192.168.201.145@tcp126 at host 192.168.201.145:988: is it running a compatible version of Lustre? [15466.973286] LNetError: Skipped 4 previous similar messages [15468.750334] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [15471.013043] LNetError: 463852:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.145 [15471.033397] LNetError: Protocol error connecting to 192.168.201.145@tcp126 at host 192.168.201.145:988: is it running a compatible version of Lustre? [15479.201825] LNetError: 463861:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.145 [15479.210759] LNetError: Protocol error connecting to 192.168.201.145@tcp126 at host 192.168.201.145:988: is it running a compatible version of Lustre? [15479.475205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15479.696661] Lustre: 463845:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15479.809053] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15479.814518] Lustre: Skipped 1 previous similar message [15479.928333] Lustre: lustre-MDT0001: new disk, initializing [15479.987107] Lustre: 464646:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.145@tcp: can't add NID 192.168.201.145@tcp130: rc = -75 [15480.001833] Lustre: 464646:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15480.268767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15480.322602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15480.337335] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15485.232444] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [15493.536536] LNetError: 463850:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.145 [15493.543430] LNetError: Protocol error connecting to 192.168.201.145@tcp101 at host 192.168.201.145:988: is it running a compatible version of Lustre? [15495.801302] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15504.084902] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15509.922053] LNetError: 463861:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.145 [15509.941196] LNetError: 463861:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [15509.956816] LNetError: Protocol error connecting to 192.168.201.145@tcp120 at host 192.168.201.145:988: is it running a compatible version of Lustre? [15509.969676] LNetError: Skipped 4 previous similar messages [15512.463058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15512.770993] Lustre: lustre-OST0000: new disk, initializing [15512.775336] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15512.780992] Lustre: 465822:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15512.918359] Lustre: 465827:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.145@tcp: can't add NID 192.168.201.145@tcp130: rc = -75 [15512.935406] Lustre: 465827:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [15513.017279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15519.501764] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [15522.378661] Lustre: 466278:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.145@tcp: can't add NID 192.168.201.145@tcp130: rc = -75 [15522.394804] Lustre: 466278:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15522.488342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15522.510635] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15522.766560] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15524.257055] LNet: 1 local NIs in recovery (showing 1): 192.168.201.145@tcp126 [15529.608750] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15538.151775] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15538.155748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15538.164269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15538.175920] Lustre: Skipped 1 previous similar message [15538.197672] LustreError: Skipped 1 previous similar message [15542.670962] Lustre: server umount lustre-OST0000 complete [15544.741975] LNetError: 463852:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.145 [15544.762340] LNetError: 463852:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [15544.771183] LNetError: Protocol error connecting to 192.168.201.145@tcp101 at host 192.168.201.145:988: is it running a compatible version of Lustre? [15544.779368] LNetError: Skipped 7 previous similar messages [15546.854257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15546.867244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15546.906985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15546.916678] Lustre: Skipped 1 previous similar message [15548.385946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15548.409221] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15553.141623] Lustre: server umount lustre-MDT0000 complete [15557.291579] LustreError: 463838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788538275 with bad export cookie 155264799184553297 [15557.303561] LustreError: 463838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15557.304915] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15557.561235] Lustre: server umount lustre-MDT0001 complete [15569.176235] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [15572.274941] Key type lgssc unregistered [15572.783632] LNet: 467734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15572.791638] LNetError: 467734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15572.798297] LNet: Removed LNI 192.168.201.145@tcp [15573.856903] LNet: Removed LNI 192.168.201.145@tcp100 [15573.859082] LNetError: 467734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15573.881644] LNetError: 467734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15573.900693] LNet: Removed LNI 192.168.201.145@tcp101 [15574.943271] LNet: Removed LNI 192.168.201.145@tcp102 [15574.945450] LNetError: 467734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15574.956739] LNetError: 467734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15574.964922] LNet: Removed LNI 192.168.201.145@tcp103 [15576.033098] LNet: Removed LNI 192.168.201.145@tcp104 [15577.055231] LNet: Removed LNI 192.168.201.145@tcp105 [15577.068678] LNetError: 467734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15577.077331] LNetError: 467734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15577.107125] LNet: Removed LNI 192.168.201.145@tcp106 [15578.143168] LNet: Removed LNI 192.168.201.145@tcp107 [15579.167192] LNet: Removed LNI 192.168.201.145@tcp108 [15580.192271] LNet: Removed LNI 192.168.201.145@tcp109 [15581.215190] LNet: Removed LNI 192.168.201.145@tcp110 [15581.219698] LNetError: 467734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15581.229551] LNetError: 467734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15581.242705] LNet: Removed LNI 192.168.201.145@tcp111 [15582.304792] LNet: Removed LNI 192.168.201.145@tcp112 [15583.327269] LNet: Removed LNI 192.168.201.145@tcp113 [15584.351743] LNet: Removed LNI 192.168.201.145@tcp114 [15585.375676] LNet: Removed LNI 192.168.201.145@tcp115 [15586.399197] LNet: Removed LNI 192.168.201.145@tcp116 [15587.424815] LNet: Removed LNI 192.168.201.145@tcp117 [15588.447287] LNet: Removed LNI 192.168.201.145@tcp118 [15589.535184] LNet: Removed LNI 192.168.201.145@tcp119 [15589.537280] LNetError: 467734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15589.563514] LNetError: 467734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15589.576475] LNet: Removed LNI 192.168.201.145@tcp120 [15590.623236] LNet: Removed LNI 192.168.201.145@tcp121 [15591.647147] LNet: Removed LNI 192.168.201.145@tcp122 [15592.671132] LNet: Removed LNI 192.168.201.145@tcp123 [15593.695278] LNet: Removed LNI 192.168.201.145@tcp124 [15594.719211] LNet: Removed LNI 192.168.201.145@tcp125 [15594.721315] LNet: Removed LNI 192.168.201.145@tcp126 [15595.743194] LNet: Removed LNI 192.168.201.145@tcp127 [15596.767256] LNet: Removed LNI 192.168.201.145@tcp128 [15597.791711] LNet: Removed LNI 192.168.201.145@tcp129 [15597.797405] LNet: Removed LNI 192.168.201.145@tcp130 [15597.801496] LNet: Removed LNI 192.168.201.145@tcp131 [15597.805929] LNet: Removed LNI 192.168.201.145@tcp132 [15597.808528] LNet: Removed LNI 192.168.201.145@tcp133 [15597.811475] LNet: Removed LNI 192.168.201.145@tcp134 [15597.828605] LNet: Removed LNI 192.168.201.145@tcp135 [15599.032513] Key type .llcrypt unregistered [15599.034126] Key type ._llcrypt unregistered [15624.672262] Key type ._llcrypt registered [15624.674339] Key type .llcrypt registered [15624.806497] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [15641.108843] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [15642.162069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15642.277515] alg: No test for adler32 (adler32-zlib) [15643.305728] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [15643.528550] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [15645.191400] Key type lgssc registered [15646.289953] Lustre: Echo OBD driver; http://www.lustre.org/ [15682.912327] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:13:18 (1788538398) [15702.016606] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [15704.612504] Key type lgssc unregistered [15704.847067] LNet: 471816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15704.851594] LNetError: 471816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15704.865693] LNet: Removed LNI 192.168.201.145@tcp [15705.459430] Key type .llcrypt unregistered [15705.460673] Key type ._llcrypt unregistered [15720.331477] Key type ._llcrypt registered [15720.332819] Key type .llcrypt registered [15720.475471] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [15721.683939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15721.838329] alg: No test for adler32 (adler32-zlib) [15722.985084] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [15723.288991] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [15724.983230] Key type lgssc registered [15726.143811] Lustre: Echo OBD driver; http://www.lustre.org/ [15735.073062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15735.099222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15736.349301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15736.381861] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15736.495022] Lustre: lustre-MDT0000: new disk, initializing [15736.614590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15736.626399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15741.472124] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [15751.682147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15751.774760] Lustre: 473024:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15751.834460] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15751.839454] Lustre: Skipped 1 previous similar message [15751.893597] Lustre: lustre-MDT0001: new disk, initializing [15751.990342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15752.030964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15752.041339] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15756.486575] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [15765.697155] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15771.134531] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15777.255639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15777.484407] Lustre: lustre-OST0000: new disk, initializing [15777.489607] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15777.497635] Lustre: 474996:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15777.574034] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15783.606826] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [15786.520092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15786.546171] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15786.644985] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15792.404481] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15796.899299] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing lnet_if_list [15802.593522] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15802.626243] LNet: Added LNI 192.168.201.145@tcp10 [8/256/0/180] [15808.292536] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15808.336934] LNet: Added LNI 192.168.201.145@tcp11 [8/256/0/180] [15812.853190] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15812.885744] LNet: Added LNI 192.168.201.145@tcp12 [8/256/0/180] [15817.237454] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15817.294752] LNet: Added LNI 192.168.201.145@tcp13 [8/256/0/180] [15823.171236] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15823.209282] LNet: Added LNI 192.168.201.145@tcp14 [8/256/0/180] [15828.608730] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15828.652249] LNet: Added LNI 192.168.201.145@tcp15 [8/256/0/180] [15834.899474] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15834.937640] LNet: Added LNI 192.168.201.145@tcp16 [8/256/0/180] [15842.523290] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15842.601532] LNet: Added LNI 192.168.201.145@tcp17 [8/256/0/180] [15850.430326] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15850.456567] LNet: Added LNI 192.168.201.145@tcp18 [8/256/0/180] [15857.633310] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15857.686357] LNet: Added LNI 192.168.201.145@tcp19 [8/256/0/180] [15865.182726] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15865.237938] LNet: Added LNI 192.168.201.145@tcp20 [8/256/0/180] [15873.855619] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15873.894546] LNet: Added LNI 192.168.201.145@tcp21 [8/256/0/180] [15878.717485] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15878.737664] LNet: Added LNI 192.168.201.145@tcp22 [8/256/0/180] [15884.365141] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15884.394398] LNet: Added LNI 192.168.201.145@tcp23 [8/256/0/180] [15891.023494] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15891.058647] LNet: Added LNI 192.168.201.145@tcp24 [8/256/0/180] [15897.729641] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15897.757367] LNet: Added LNI 192.168.201.145@tcp25 [8/256/0/180] [15903.575670] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15903.594955] LNet: Added LNI 192.168.201.145@tcp26 [8/256/0/180] [15910.317069] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15910.341249] LNet: Added LNI 192.168.201.145@tcp27 [8/256/0/180] [15918.390210] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15918.430950] LNet: Added LNI 192.168.201.145@tcp28 [8/256/0/180] [15925.101198] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15925.125353] LNet: Added LNI 192.168.201.145@tcp29 [8/256/0/180] [15931.339771] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15931.357700] LNet: Added LNI 192.168.201.145@tcp30 [8/256/0/180] [15938.191529] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15938.248754] LNet: Added LNI 192.168.201.145@tcp31 [8/256/0/180] [15945.998617] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15946.076712] LNet: Added LNI 192.168.201.145@tcp32 [8/256/0/180] [15954.477185] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15954.511726] LNet: Added LNI 192.168.201.145@tcp33 [8/256/0/180] [15964.688837] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15964.730418] LNet: Added LNI 192.168.201.145@tcp34 [8/256/0/180] [15973.136421] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15973.187772] LNet: Added LNI 192.168.201.145@tcp35 [8/256/0/180] [15980.706730] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15980.728879] LNet: Added LNI 192.168.201.145@tcp36 [8/256/0/180] [15988.627336] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15988.647230] LNet: Added LNI 192.168.201.145@tcp37 [8/256/0/180] [15994.231736] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15994.299643] LNet: Added LNI 192.168.201.145@tcp38 [8/256/0/180] [16001.108426] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [16001.159431] LNet: Added LNI 192.168.201.145@tcp39 [8/256/0/180] [16008.521510] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16008.548811] LNet: Added LNI 192.168.201.145@tcp40 [8/256/0/180] [16016.010172] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16016.053875] LNet: Added LNI 192.168.201.145@tcp41 [8/256/0/180] [16023.899251] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16023.962888] LNet: Added LNI 192.168.201.145@tcp42 [8/256/0/180] [16033.688646] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16033.720941] LNet: Added LNI 192.168.201.145@tcp43 [8/256/0/180] [16042.201900] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16042.218612] LNet: Added LNI 192.168.201.145@tcp44 [8/256/0/180] [16051.112265] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16051.136447] LNet: Added LNI 192.168.201.145@tcp45 [8/256/0/180] [16059.803438] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16059.839950] LNet: Added LNI 192.168.201.145@tcp46 [8/256/0/180] [16067.251723] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16067.271985] LNet: Added LNI 192.168.201.145@tcp47 [8/256/0/180] [16075.010809] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16075.033381] LNet: Added LNI 192.168.201.145@tcp48 [8/256/0/180] [16083.138861] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16083.161255] LNet: Added LNI 192.168.201.145@tcp49 [8/256/0/180] [16090.262429] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16090.294339] LNet: Added LNI 192.168.201.145@tcp50 [8/256/0/180] [16106.246509] Lustre: server umount lustre-MDT0000 complete [16109.484244] LustreError: 473821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788538827 with bad export cookie 14486726385286852482 [16109.500452] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16109.536181] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16109.545265] Lustre: Skipped 1 previous similar message [16109.548969] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [16109.852370] Lustre: server umount lustre-MDT0001 complete [16126.419978] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [16134.550070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16153.638336] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc90b30e4b94d3976 [16153.672930] Lustre: MGC192.168.201.145@tcp: Connection restored to 0@lo (at 0@lo) [16154.090063] LustreError: 474990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16154.124348] LustreError: 474990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [16154.174121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16159.202213] LustreError: 474991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16159.359524] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [16164.328494] LustreError: 474990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16169.441571] LustreError: 474991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16169.519245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16170.017416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16175.186940] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [16176.119845] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16183.722421] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16189.734319] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16215.521604] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16215.525909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16215.549481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16216.038315] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16217.071060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16217.082647] Lustre: Skipped 2 previous similar messages [16219.391696] Lustre: server umount lustre-OST0000 complete [16226.280823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16226.293878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16226.316507] Lustre: Skipped 1 previous similar message [16226.324678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16230.519858] Lustre: server umount lustre-MDT0000 complete [16232.428583] LustreError: 499243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16232.447772] LustreError: 499243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16234.766172] LustreError: 473019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788538953 with bad export cookie 14486726385286855030 [16234.769043] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16234.776805] LustreError: 473019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16235.177636] Lustre: server umount lustre-MDT0001 complete [16247.542324] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [16250.889659] Key type lgssc unregistered [16251.445985] LNet: 502043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16251.461505] LNetError: 502043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16251.482863] LNet: Removed LNI 192.168.201.145@tcp [16251.484648] LNet: Removed LNI 192.168.201.145@tcp10 [16251.499771] LNet: Removed LNI 192.168.201.145@tcp11 [16251.501783] LNet: Removed LNI 192.168.201.145@tcp12 [16251.513643] LNet: Removed LNI 192.168.201.145@tcp13 [16251.515572] LNet: Removed LNI 192.168.201.145@tcp14 [16251.523486] LNet: Removed LNI 192.168.201.145@tcp15 [16251.527447] LNet: Removed LNI 192.168.201.145@tcp16 [16251.530394] LNet: Removed LNI 192.168.201.145@tcp17 [16251.533424] LNet: Removed LNI 192.168.201.145@tcp18 [16251.536541] LNet: Removed LNI 192.168.201.145@tcp19 [16251.539940] LNet: Removed LNI 192.168.201.145@tcp20 [16251.541908] LNet: Removed LNI 192.168.201.145@tcp21 [16251.544179] LNet: Removed LNI 192.168.201.145@tcp22 [16251.545920] LNet: Removed LNI 192.168.201.145@tcp23 [16251.549490] LNet: Removed LNI 192.168.201.145@tcp24 [16251.551901] LNet: Removed LNI 192.168.201.145@tcp25 [16251.556125] LNet: Removed LNI 192.168.201.145@tcp26 [16251.558982] LNet: Removed LNI 192.168.201.145@tcp27 [16251.561280] LNet: Removed LNI 192.168.201.145@tcp28 [16251.563869] LNet: Removed LNI 192.168.201.145@tcp29 [16251.566354] LNet: Removed LNI 192.168.201.145@tcp30 [16251.568957] LNet: Removed LNI 192.168.201.145@tcp31 [16251.571683] LNet: Removed LNI 192.168.201.145@tcp32 [16251.574289] LNet: Removed LNI 192.168.201.145@tcp33 [16251.576556] LNet: Removed LNI 192.168.201.145@tcp34 [16251.579363] LNet: Removed LNI 192.168.201.145@tcp35 [16251.581933] LNet: Removed LNI 192.168.201.145@tcp36 [16251.584753] LNet: Removed LNI 192.168.201.145@tcp37 [16251.587253] LNet: Removed LNI 192.168.201.145@tcp38 [16251.589781] LNet: Removed LNI 192.168.201.145@tcp39 [16251.592486] LNet: Removed LNI 192.168.201.145@tcp40 [16251.595399] LNet: Removed LNI 192.168.201.145@tcp41 [16251.597858] LNet: Removed LNI 192.168.201.145@tcp42 [16251.601987] LNet: Removed LNI 192.168.201.145@tcp43 [16251.604334] LNet: Removed LNI 192.168.201.145@tcp44 [16251.606927] LNet: Removed LNI 192.168.201.145@tcp45 [16251.609424] LNet: Removed LNI 192.168.201.145@tcp46 [16251.611563] LNet: Removed LNI 192.168.201.145@tcp47 [16251.614608] LNet: Removed LNI 192.168.201.145@tcp48 [16251.616968] LNet: Removed LNI 192.168.201.145@tcp49 [16251.626173] LNet: Removed LNI 192.168.201.145@tcp50 [16252.603056] Key type .llcrypt unregistered [16252.606894] Key type ._llcrypt unregistered [16277.839319] Key type ._llcrypt registered [16277.840943] Key type .llcrypt registered [16278.082984] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [16296.445990] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [16298.239659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16298.282586] alg: No test for adler32 (adler32-zlib) [16299.269689] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [16299.497256] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [16301.161737] Key type lgssc registered [16302.199919] Lustre: Echo OBD driver; http://www.lustre.org/ [16338.001416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16338.061332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16339.288469] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16339.332729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16339.460739] Lustre: lustre-MDT0000: new disk, initializing [16339.551762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16339.578204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16343.766720] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [16354.473167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16354.646519] Lustre: 505250:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16354.709295] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16354.719552] Lustre: Skipped 1 previous similar message [16354.824244] Lustre: lustre-MDT0001: new disk, initializing [16355.002778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16355.042543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16355.060295] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16360.352835] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [16370.946536] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16377.009444] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16383.902858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16384.129230] Lustre: *** cfs_fail_loc=90f, val=0*** [16384.199806] Lustre: lustre-OST0000: new disk, initializing [16384.207613] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16384.229599] Lustre: 507266:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16384.331837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16390.696478] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [16392.767858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16392.775460] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16392.911460] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16399.773556] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16408.544242] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16408.557690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16408.560537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16408.563668] Lustre: Skipped 1 previous similar message [16408.574217] Lustre: Skipped 1 previous similar message [16413.668684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16413.672179] Lustre: Skipped 1 previous similar message [16418.791828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16418.795460] Lustre: Skipped 1 previous similar message [16420.831437] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16421.012184] Lustre: server umount lustre-OST0000 complete [16426.977549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16426.988459] LustreError: Skipped 1 previous similar message [16426.994986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16427.011888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16429.039808] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16430.748338] Lustre: server umount lustre-MDT0000 complete [16434.648111] LustreError: 506046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788539153 with bad export cookie 4506002908353807937 [16434.653051] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16434.656306] LustreError: 506046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16435.199907] Lustre: server umount lustre-MDT0001 complete [16444.508643] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [16446.826497] Key type lgssc unregistered [16447.091926] LNet: 509172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16447.099890] LNetError: 509172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16448.169592] LNet: Removed LNI 192.168.201.145@tcp [16448.986583] Key type .llcrypt unregistered [16448.988874] Key type ._llcrypt unregistered [16465.084167] Key type ._llcrypt registered [16465.085802] Key type .llcrypt registered [16465.205405] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [16466.293567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16466.302150] alg: No test for adler32 (adler32-zlib) [16467.263913] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [16467.409424] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [16469.050090] Key type lgssc registered [16470.034133] Lustre: Echo OBD driver; http://www.lustre.org/ [16477.298258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16477.316887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16478.796155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16483.444909] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [16488.930707] LustreError: 510420:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16488.965618] LustreError: 510420:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [16491.864817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16492.315427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16496.524585] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [16504.623396] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16511.943821] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16519.450738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16519.706442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16524.668351] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [16532.551182] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16546.558777] loop0: detected capacity change from 0 to 204800000 [16546.793624] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16546.804856] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16546.815837] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16552.106915] loop0: detected capacity change from 0 to 204800000 [16552.305325] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16552.314213] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16552.323110] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16557.474183] loop0: detected capacity change from 0 to 204800000 [16562.518385] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16562.661174] Lustre: fs15246-OST0000: new disk, initializing [16562.664750] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16562.671571] Lustre: 514012:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16562.763523] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16568.562207] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [16578.674521] loop1: detected capacity change from 0 to 204800000 [16585.662104] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16585.761072] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16585.866400] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16585.967456] Lustre: fs15246-MDT0000: new disk, initializing [16586.152219] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16586.183155] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16586.281092] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16587.363588] Lustre: 515134:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000021ff2e58 x1875419264497792/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788539321 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16591.992961] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [16596.535834] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16596.544117] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16596.592342] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16614.890307] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16614.914511] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16618.940776] Lustre: server umount fs15246-MDT0000 complete [16620.026978] LustreError: 511466:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16623.078104] LustreError: 510416:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16628.254130] Lustre: server umount fs15246-OST0000 complete [16639.968666] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16639.989220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16640.013904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16642.018228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16642.030402] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16642.054301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16648.681894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16648.689358] Lustre: Skipped 3 previous similar messages [16653.801136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16653.810143] Lustre: Skipped 1 previous similar message [16654.303299] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16654.453462] Lustre: server umount lustre-OST0000 complete [16658.916441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16664.032282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16664.039020] Lustre: Skipped 3 previous similar messages [16664.638986] Lustre: server umount lustre-MDT0000 complete [16668.633202] LustreError: 511153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788539387 with bad export cookie 2514592754885815149 [16668.635492] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16668.664076] LustreError: 511153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16669.070278] Lustre: server umount lustre-MDT0001 complete [16680.059839] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [16682.583019] Key type lgssc unregistered [16682.901736] LNet: 518233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16682.910487] LNetError: 518233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16682.941847] LNet: Removed LNI 192.168.201.145@tcp [16683.725799] Key type .llcrypt unregistered [16683.727562] Key type ._llcrypt unregistered [16704.669968] Key type ._llcrypt registered [16704.676349] Key type .llcrypt registered [16704.840771] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [16706.790724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16706.859104] alg: No test for adler32 (adler32-zlib) [16708.120157] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [16708.441861] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [16710.172479] Key type lgssc registered [16711.880963] Lustre: Echo OBD driver; http://www.lustre.org/ [16753.812892] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [16769.446549] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [16806.721281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16806.751763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16807.970692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16808.000596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16808.079118] Lustre: lustre-MDT0000: new disk, initializing [16808.166528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16808.188284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16812.902594] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [16824.140888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16824.245377] Lustre: 522712:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16824.313426] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16824.316961] Lustre: Skipped 1 previous similar message [16824.383491] Lustre: lustre-MDT0001: new disk, initializing [16824.487147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16824.529024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16824.546109] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16829.116134] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [16839.181900] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16844.162983] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16850.428343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16850.809461] Lustre: lustre-OST0000: new disk, initializing [16850.819551] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16850.833959] Lustre: 524682:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16850.955030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16857.935659] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [16858.210373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16858.224509] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16858.367872] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16867.728745] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16873.299951] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16873.485632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16878.374514] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16878.617792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16884.197359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16884.209303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16884.218381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16884.220622] Lustre: Skipped 1 previous similar message [16884.231702] Lustre: Skipped 1 previous similar message [16887.075896] Lustre: server umount lustre-OST0000 complete [16891.360950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16891.363927] LustreError: Skipped 1 previous similar message [16891.370957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16891.387084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16894.448223] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16894.456481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16894.486992] Lustre: Skipped 1 previous similar message [16897.341264] Lustre: server umount lustre-MDT0000 complete [16901.099407] LustreError: 522707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788539619 with bad export cookie 9287678626547039616 [16901.105064] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16901.113011] LustreError: 522707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16901.363460] Lustre: server umount lustre-MDT0001 complete [16927.425946] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [16940.328030] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [16976.766576] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16597 sec ========= 12:34:52 (1788539692) [16978.415972] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:34:54 (1788539694) === [16981.963740] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:34:58 (1788539698) === [17019.386611] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [17022.251366] Key type lgssc unregistered [17022.555967] LNet: 531327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17022.562374] LNetError: 531327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17022.582647] LNet: Removed LNI 192.168.201.145@tcp [17023.690430] Key type .llcrypt unregistered [17023.692431] Key type ._llcrypt unregistered