[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 464613758 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002302] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.006582] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007004] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009008] pid_max: default: 32768 minimum: 301 [ 0.010133] LSM: Security Framework initializing [ 0.011037] Yama: becoming mindful. [ 0.012028] SELinux: Initializing. [ 0.013064] *** VALIDATE selinux *** [ 0.021080] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025117] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026142] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028114] *** VALIDATE tmpfs *** [ 0.029435] *** VALIDATE proc *** [ 0.030248] *** VALIDATE cgroup *** [ 0.031009] *** VALIDATE cgroup2 *** [ 0.032254] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034086] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036023] Spectre V2 : User space: Vulnerable [ 0.037007] Speculative Store Bypass: Vulnerable [ 0.040003] debug: unmapping init [mem 0xffffffffab659000-0xffffffffab660fff] [ 0.042914] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043620] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044020] ... version: 2 [ 0.045010] ... bit width: 48 [ 0.046010] ... generic registers: 4 [ 0.046902] ... value mask: 0000ffffffffffff [ 0.047012] ... max period: 00007fffffffffff [ 0.048010] ... fixed-purpose events: 3 [ 0.048895] ... event mask: 000000070000000f [ 0.049307] rcu: Hierarchical SRCU implementation. [ 0.051436] smp: Bringing up secondary CPUs ... [ 0.052554] x86: Booting SMP configuration: [ 0.053020] .... node #0, CPUs: #1 #2 #3 [ 0.060247] smp: Brought up 1 node, 4 CPUs [ 0.061529] smpboot: Max logical packages: 1 [ 0.062007] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.100089] node 0 deferred pages initialised in 36ms [ 0.103216] devtmpfs: initialized [ 0.104174] x86/mm: Memory block size: 128MB [ 0.106601] gcov: version magic: 0x41383552 [ 0.109289] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.113099] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.116844] pinctrl core: initialized pinctrl subsystem [ 0.120260] [ 0.121009] ************************************************************* [ 0.124016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.127026] ** ** [ 0.130075] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.133013] ** ** [ 0.137020] ** This means that this kernel is built to expose internal ** [ 0.140014] ** IOMMU data structures, which may compromise security on ** [ 0.143015] ** your system. ** [ 0.146015] ** ** [ 0.148015] ** If you see this message and you are not debugging the ** [ 0.151014] ** kernel, report this immediately to your vendor! ** [ 0.153011] ** ** [ 0.156014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158010] ************************************************************* [ 0.161594] NET: Registered protocol family 16 [ 0.163506] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.166061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.169057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.173011] cpuidle: using governor menu [ 0.174775] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.175462] PCI: Using configuration type 1 for base access [ 0.176141] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.183148] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.184037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.186081] cryptd: max_cpu_qlen set to 1000 [ 0.189324] ACPI: Added _OSI(Module Device) [ 0.191014] ACPI: Added _OSI(Processor Device) [ 0.193015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.195013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.200361] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.207447] ACPI: Interpreter enabled [ 0.209060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.211013] ACPI: Using IOAPIC for interrupt routing [ 0.212110] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.216503] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.227903] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.230042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.233019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.237080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.242000] acpiphp: Slot [2] registered [ 0.242000] acpiphp: Slot [5] registered [ 0.244116] acpiphp: Slot [6] registered [ 0.246144] acpiphp: Slot [7] registered [ 0.248114] acpiphp: Slot [8] registered [ 0.249100] acpiphp: Slot [9] registered [ 0.250142] acpiphp: Slot [10] registered [ 0.252090] acpiphp: Slot [3] registered [ 0.254101] acpiphp: Slot [4] registered [ 0.255000] acpiphp: Slot [11] registered [ 0.257089] acpiphp: Slot [12] registered [ 0.258094] acpiphp: Slot [13] registered [ 0.260101] acpiphp: Slot [14] registered [ 0.261100] acpiphp: Slot [15] registered [ 0.263136] acpiphp: Slot [16] registered [ 0.264100] acpiphp: Slot [17] registered [ 0.265097] acpiphp: Slot [18] registered [ 0.266092] acpiphp: Slot [19] registered [ 0.268111] acpiphp: Slot [20] registered [ 0.269099] acpiphp: Slot [21] registered [ 0.271117] acpiphp: Slot [22] registered [ 0.272080] acpiphp: Slot [23] registered [ 0.273086] acpiphp: Slot [24] registered [ 0.275070] acpiphp: Slot [25] registered [ 0.276060] acpiphp: Slot [26] registered [ 0.277040] acpiphp: Slot [27] registered [ 0.278000] acpiphp: Slot [28] registered [ 0.278000] acpiphp: Slot [29] registered [ 0.280099] acpiphp: Slot [30] registered [ 0.282089] acpiphp: Slot [31] registered [ 0.283078] PCI host bridge to bus 0000:00 [ 0.284013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.286026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.288018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.291022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.293014] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.295025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.297129] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.299550] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.303261] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.315016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.322041] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.327020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.330035] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.333024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.336575] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.338793] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.342048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.344828] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.350012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.362015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.365850] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.369633] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.380016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.386016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.407019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.415601] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.422013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.433020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.455022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.462327] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.469027] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.475029] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.489029] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.497000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.504016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.511017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.525021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.534829] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.542024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.547022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.570016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.593101] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.598016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.603014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.616017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.623815] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.626305] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.627310] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.630372] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.631137] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.635033] iommu: Default domain type: Passthrough [ 0.636301] SCSI subsystem initialized [ 0.637106] ACPI: bus type USB registered [ 0.640130] usbcore: registered new interface driver usbfs [ 0.641060] usbcore: registered new interface driver hub [ 0.643066] usbcore: registered new device driver usb [ 0.644126] pps_core: LinuxPPS API ver. 1 registered [ 0.645011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.647051] PTP clock support registered [ 0.649082] EDAC MC: Ver: 3.0.0 [ 0.650331] PCI: Using ACPI for IRQ routing [ 0.651614] NetLabel: Initializing [ 0.653012] NetLabel: domain hash size = 128 [ 0.654006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.655062] NetLabel: unlabeled traffic allowed by default [ 0.657106] vgaarb: loaded [ 0.658170] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.659010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.666830] clocksource: Switched to clocksource kvm-clock [ 0.757965] VFS: Disk quotas dquot_6.6.0 [ 0.759467] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.761614] *** VALIDATE ramfs *** [ 0.762522] *** VALIDATE hugetlbfs *** [ 0.763727] pnp: PnP ACPI init [ 0.765524] pnp: PnP ACPI: found 6 devices [ 0.782900] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.785737] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.787789] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.789831] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.792085] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.794691] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.797132] NET: Registered protocol family 2 [ 0.799615] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.804694] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.807759] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.812210] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.814925] TCP: Hash tables configured (established 65536 bind 65536) [ 0.817100] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.819358] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.821511] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.823687] NET: Registered protocol family 1 [ 0.825440] RPC: Registered named UNIX socket transport module. [ 0.827689] RPC: Registered udp transport module. [ 0.829463] RPC: Registered tcp transport module. [ 0.831645] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.834481] NET: Registered protocol family 44 [ 0.836571] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.838298] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.840163] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.842211] PCI: CLS 0 bytes, default 64 [ 0.843942] Unpacking initramfs... [ 3.338242] debug: unmapping init [mem 0xffff9ccbbcc54000-0xffff9ccbbffbffff] [ 3.344268] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.347283] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.351439] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.669517] Initialise system trusted keyrings [ 4.672164] Key type blacklist registered [ 4.674498] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.686321] zbud: loaded [ 4.689664] *** VALIDATE nfs *** [ 4.691278] *** VALIDATE nfs4 *** [ 4.693171] pstore: using deflate compression [ 4.697466] Platform Keyring initialized [ 5.104850] NET: Registered protocol family 38 [ 5.107278] Key type asymmetric registered [ 5.109201] Asymmetric key parser 'x509' registered [ 5.110940] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.113377] io scheduler mq-deadline registered [ 5.114898] io scheduler kyber registered [ 5.116884] io scheduler bfq registered [ 5.119790] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.123547] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.127347] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.131297] ACPI: Power Button [PWRF] [ 5.141143] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.152844] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.242353] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.266879] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.327771] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.378225] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.415234] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.427174] Non-volatile memory driver v1.3 [ 5.430175] Linux agpgart interface v0.103 [ 5.488656] virtio_blk virtio1: [vda] 149792 512-byte logical blocks (76.7 MB/73.1 MiB) [ 5.491401] vda: detected capacity change from 0 to 76693504 [ 5.515453] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.519570] vdb: detected capacity change from 0 to 1073741824 [ 5.554276] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.558047] vdc: detected capacity change from 0 to 2621440000 [ 5.614934] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.620154] vdd: detected capacity change from 0 to 2621440000 [ 5.636292] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.639825] vde: detected capacity change from 0 to 4294967296 [ 5.698306] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.712374] vdf: detected capacity change from 0 to 4294967296 [ 5.740449] libphy: Fixed MDIO Bus: probed [ 5.771712] usbcore: registered new interface driver usbserial_generic [ 5.779206] usbserial: USB Serial support registered for generic [ 5.783221] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.796194] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.801871] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.812027] mousedev: PS/2 mouse device common for all mice [ 5.815957] rtc_cmos 00:05: RTC can wake from S4 [ 5.823956] rtc_cmos 00:05: registered as rtc0 [ 5.827525] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.828614] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.830915] intel_pstate: CPU model not supported [ 5.835404] hid: raw HID events driver (C) Jiri Kosina [ 5.856800] usbcore: registered new interface driver usbhid [ 5.860269] usbhid: USB HID core driver [ 5.864700] drop_monitor: Initializing network drop monitor service [ 5.868632] Initializing XFRM netlink socket [ 5.875098] NET: Registered protocol family 10 [ 5.877618] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.888222] Segment Routing with IPv6 [ 5.892213] NET: Registered protocol family 17 [ 5.894675] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.902505] mpls_gso: MPLS GSO support [ 5.913388] RAS: Correctable Errors collector initialized. [ 5.917052] AVX version of gcm_enc/dec engaged. [ 5.921230] AES CTR mode by8 optimization enabled [ 6.358167] sched_clock: Marking stable (6358137443, 0)->(7197343525, -839206082) [ 6.382279] registered taskstats version 1 [ 6.391222] Loading compiled-in X.509 certificates [ 6.398465] zswap: loaded using pool lzo/zbud [ 6.536648] Key type big_key registered [ 6.563653] Key type encrypted registered [ 6.567534] ima: No TPM chip found, activating TPM-bypass! [ 6.572819] ima: Allocated hash algorithm: sha1 [ 6.577875] ima: No architecture policies found [ 6.580763] evm: Initialising EVM extended attributes: [ 6.582530] evm: security.selinux [ 6.584487] evm: security.ima [ 6.586866] evm: security.capability [ 6.589301] evm: HMAC attrs: 0x1 [ 6.598561] rtc_cmos 00:05: setting system clock to 2026-09-03 20:46:20 UTC (1788468380) [ 6.612653] debug: unmapping init [mem 0xffffffffac603000-0xffffffffac7fffff] [ 6.618527] debug: unmapping init [mem 0xffffffffab382000-0xffffffffab658fff] [ 6.627456] Write protecting the kernel read-only data: 28672k [ 6.632679] debug: unmapping init [mem 0xffffffffa9a03000-0xffffffffa9bfffff] [ 6.638088] debug: unmapping init [mem 0xffffffffaa314000-0xffffffffaa3fffff] [ 6.751533] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.769293] systemd[1]: Detected virtualization kvm. [ 6.776120] systemd[1]: Detected architecture x86-64. [ 6.778920] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.866254] systemd[1]: No hostname configured. [ 6.868149] systemd[1]: Set hostname to . [ 6.872906] random: systemd: uninitialized urandom read (16 bytes read) [ 6.880114] systemd[1]: Initializing machine ID from random generator. [ 7.250400] random: systemd: uninitialized urandom read (16 bytes read) [ 7.253279] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 7.258213] random: systemd: uninitialized urandom read (16 bytes read) [ 7.264956] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 7.275101] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.260943] device-mapper: uevent: version 1.0.3 [ 9.272446] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 11.792989] virtio_net virtio0 ens2: renamed from eth0 [ 11.933096] random: fast init done [ 12.024519] scsi host0: ata_piix [ 12.103383] scsi host1: ata_piix [ 12.106337] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.110686] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 17.124295] random: crng init done [ 17.134245] random: 7 urandom warning(s) missed due to ratelimiting [ 20.020078] dracut-initqueue[595]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 21.686322] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.241131] printk: systemd: 26 output lines suppressed due to ratelimiting [ 25.086206] SELinux: Disabled at runtime. [ 25.216312] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.245784] systemd[1]: Detected virtualization kvm. [ 25.249412] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 27.590885] systemd[1]: initrd-switch-root.service: Succeeded. [ 27.603440] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 27.628316] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 27.635965] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 27.641346] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 27.655758] systemd[1]: Starting Journal Service... Starting Journal Service... [ 27.663489] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ 28.021819] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 29.229416] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 30.398093] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 30.579074] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 31.074957] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 31.163685] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 35.795405] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 36.410994] NFS: Registering the id_resolver key type [ 36.412480] Key type id_resolver registered [ 36.413506] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg235-server login: [ 95.955535] libcfs: loading out-of-tree module taints kernel. [ 95.998303] Key type ._llcrypt registered [ 96.000520] Key type .llcrypt registered [ 96.094493] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 115.780718] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 117.103335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 117.120462] alg: No test for adler32 (adler32-zlib) [ 118.313846] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 118.941290] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 120.753043] Key type lgssc registered [ 123.111483] Lustre: Echo OBD driver; http://www.lustre.org/ [ 140.446537] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 189.429711] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 193.349174] hrtimer: interrupt took 8161601 ns [ 201.182508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 201.242604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.498295] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 202.517505] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 202.637757] Lustre: lustre-MDT0000: new disk, initializing [ 202.740406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.768035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 206.745564] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.901510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 219.984899] Lustre: 6521:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 220.032050] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 220.035698] Lustre: Skipped 1 previous similar message [ 220.137305] Lustre: lustre-MDT0001: new disk, initializing [ 220.317335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 220.386177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 220.404124] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 225.417995] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.129323] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 243.858301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 244.262964] Lustre: lustre-OST0000: new disk, initializing [ 244.270581] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 244.283963] Lustre: 8459:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 244.399058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 250.621204] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 254.002784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 254.012522] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 254.106982] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 267.042821] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 267.243905] Lustre: lustre-OST0001: new disk, initializing [ 267.247191] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 267.258967] Lustre: 9533:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 267.365445] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 273.427463] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 274.073076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 274.090052] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 279.537421] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 284.583702] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 290.240480] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 296.170090] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing check_logdir /tmp/testlogs/ [ 299.427646] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing yml_node [ 303.946860] Lustre: DEBUG MARKER: Client: 2.17.58.4 [ 306.296156] Lustre: DEBUG MARKER: MDS: 2.17.58.4 [ 308.053826] Lustre: DEBUG MARKER: OSS: 2.17.58.4 [ 309.558204] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 16:51:22 EDT 2026 [ 326.349460] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 327.957962] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 343.007738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 343.018419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.040878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 346.093146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 346.096904] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 346.118331] Lustre: Skipped 2 previous similar messages [ 348.927486] Lustre: server umount lustre-MDT0000 complete [ 351.211870] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 351.245528] Lustre: 12360:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 356.327474] Lustre: 12359:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 356.349683] LustreError: 6529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 356.359467] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 356.368695] LustreError: 6529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 356.389251] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 356.394682] Lustre: Skipped 3 previous similar messages [ 359.041324] Lustre: server umount lustre-MDT0001 complete [ 361.440218] Lustre: 12359:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 363.031811] Lustre: server umount lustre-OST0000 complete [ 366.547617] Lustre: server umount lustre-OST0001 complete [ 372.715209] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 380.864911] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 419.305275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 419.601453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 419.636628] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 419.714665] Lustre: lustre-MDT0000: new disk, initializing [ 419.844626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 419.868191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 424.428497] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 434.196567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 434.279363] Lustre: 15085:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 434.357340] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 434.360922] Lustre: Skipped 1 previous similar message [ 434.452294] Lustre: lustre-MDT0001: new disk, initializing [ 434.573914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 434.634523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 434.649255] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 438.846555] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 447.474929] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 454.046102] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 462.253095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 462.699784] Lustre: lustre-OST0000: new disk, initializing [ 462.707202] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 462.721214] Lustre: 17057:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 462.909283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 464.367105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 464.375900] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 464.476110] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 470.535386] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 482.943698] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 488.501439] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 488.700022] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 494.635863] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 494.791422] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 500.206691] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 500.209428] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 500.219382] LustreError: Skipped 1 previous similar message [ 500.231414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 500.238830] Lustre: Skipped 2 previous similar messages [ 500.260699] Lustre: Skipped 1 previous similar message [ 503.378588] Lustre: server umount lustre-OST0000 complete [ 510.434525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.443523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 510.449072] Lustre: Skipped 1 previous similar message [ 510.467742] Lustre: Skipped 1 previous similar message [ 520.671808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 520.680864] Lustre: Skipped 3 previous similar messages [ 522.207630] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 522.538152] Lustre: server umount lustre-MDT0000 complete [ 525.798441] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 525.801281] LustreError: 15092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.829418] Lustre: 18675:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 525.843816] LustreError: 15092:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 527.135643] Lustre: server umount lustre-MDT0001 complete [ 531.551523] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 16:55:04 (1788468904) [ 534.212311] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 536.684616] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 16:55:08 (1788468908) [ 545.614454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 545.952195] LustreError: 19473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.011380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 550.015949] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 551.401719] LustreError: 19474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 555.497979] LustreError: 19473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.042195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 562.755249] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 569.703811] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 574.639396] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 580.632614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 587.406911] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 594.773111] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 604.659374] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 604.676738] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 604.714225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 608.855593] Lustre: server umount lustre-OST0000 complete [ 614.886524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 614.899174] LustreError: Skipped 1 previous similar message [ 619.593611] Lustre: server umount lustre-MDT0000 complete [ 621.039674] LustreError: 19473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 624.418609] LustreError: 19454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788468998 with bad export cookie 5009683892894450455 [ 624.426647] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 624.444391] LustreError: 19454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 624.487848] Lustre: 22318:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 625.024492] Lustre: server umount lustre-MDT0001 complete [ 640.071546] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 645.190882] Key type lgssc unregistered [ 645.760466] LNet: 23009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 645.763826] LNetError: 23009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 645.815371] LNet: Removed LNI 192.168.202.135@tcp [ 647.657196] Key type .llcrypt unregistered [ 647.665831] Key type ._llcrypt unregistered [ 675.798055] Key type ._llcrypt registered [ 675.799628] Key type .llcrypt registered [ 675.957712] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 677.967860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 678.003212] alg: No test for adler32 (adler32-zlib) [ 679.082335] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 679.271143] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 680.927388] Key type lgssc registered [ 682.439312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 692.499485] loop: module loaded [ 692.627030] loop0: detected capacity change from 0 to 204800000 [ 692.780987] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 692.793783] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 693.247767] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 700.315490] loop0: detected capacity change from 0 to 204800000 [ 700.722535] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 700.740305] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 700.788281] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 708.375284] loop0: detected capacity change from 0 to 204800000 [ 715.539550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 715.581972] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 717.166870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 717.312780] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 717.438432] Lustre: lustre-MDT0000: new disk, initializing [ 717.641757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 717.675938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 722.471593] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 731.966611] loop1: detected capacity change from 0 to 204800000 [ 738.957860] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 739.336874] Lustre: lustre-OST07c6: new disk, initializing [ 739.341604] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 739.346612] Lustre: Skipped 1 previous similar message [ 739.355966] Lustre: 26202:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 739.452051] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 745.135057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 745.152279] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 745.228056] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 747.094726] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 752.818236] Lustre: 25191:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 752.830977] Lustre: 25191:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 752.838363] Lustre: 25191:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 752.853392] Lustre: 25191:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 752.860440] Lustre: 25191:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 752.873137] Lustre: 25191:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 772.045492] Lustre: 26718:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 772.065136] Lustre: 26718:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 772.077776] Lustre: 26718:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 772.087209] Lustre: 26718:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 772.099163] Lustre: 26718:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 772.107292] Lustre: 26718:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 777.390463] Lustre: 25190:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 777.398371] Lustre: 25190:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 777.403813] Lustre: 25190:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 777.408760] Lustre: 25190:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.412991] Lustre: 25190:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 777.417910] Lustre: 25190:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.422180] Lustre: 25190:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 777.427869] Lustre: 25190:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.433305] Lustre: 25190:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 777.437921] Lustre: 25190:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.447139] Lustre: 25190:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 777.460452] Lustre: 25190:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 781.600097] Lustre: 25191:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 781.608289] Lustre: 25191:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 781.614387] Lustre: 25191:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 781.620902] Lustre: 25191:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 781.626905] Lustre: 25191:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 781.632777] Lustre: 25191:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 781.638455] Lustre: 25191:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 781.645398] Lustre: 25191:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 781.651146] Lustre: 25191:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 781.657134] Lustre: 25191:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 781.661842] Lustre: 25191:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 781.669751] Lustre: 25191:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.795619] Lustre: 25189:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 789.813694] Lustre: 25189:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 789.827958] Lustre: 25189:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 789.842653] Lustre: 25189:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 789.852362] Lustre: 25189:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 789.859715] Lustre: 25189:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 789.871905] Lustre: 25189:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 789.882790] Lustre: 25189:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 789.894280] Lustre: 25189:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 789.918073] Lustre: 25189:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 789.935656] Lustre: 25189:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 789.952366] Lustre: 25189:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 799.803239] Lustre: 25191:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 799.823596] Lustre: 25191:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 799.840673] Lustre: 25191:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 799.865157] Lustre: 25191:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 799.884859] Lustre: 25191:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 799.917309] Lustre: 25191:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 799.947593] Lustre: 25191:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 799.966908] Lustre: 25191:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 799.985441] Lustre: 25191:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 799.995672] Lustre: 25191:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 800.008577] Lustre: 25191:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 800.021169] Lustre: 25191:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 847.840946] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 847.858703] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 847.879408] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 852.851128] Lustre: server umount lustre-OST07c6 complete [ 862.962046] Lustre: server umount lustre-MDT0000 complete [ 880.128208] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 882.979720] Key type lgssc unregistered [ 883.440109] LNet: 28756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.446754] LNetError: 28756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.464808] LNet: Removed LNI 192.168.202.135@tcp [ 884.251582] Key type .llcrypt unregistered [ 884.253942] Key type ._llcrypt unregistered [ 907.410132] Key type ._llcrypt registered [ 907.412528] Key type .llcrypt registered [ 907.494433] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 908.914480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 909.126790] alg: No test for adler32 (adler32-zlib) [ 910.320956] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 910.725772] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 912.431707] Key type lgssc registered [ 913.699522] Lustre: Echo OBD driver; http://www.lustre.org/ [ 922.728793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 922.743402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 924.385192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 929.471921] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 934.887018] LustreError: 30041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 934.915840] LustreError: 30041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 938.485672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 938.907232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 944.003793] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 951.775736] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 958.833665] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 967.107303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 967.465441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 969.534544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 974.649390] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 983.243801] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1004.515373] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1004.529041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1004.548531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1007.225524] Lustre: server umount lustre-OST0000 complete [ 1014.767240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1014.776301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1014.790814] Lustre: Skipped 2 previous similar messages [ 1014.806164] Lustre: Skipped 2 previous similar messages [ 1018.749746] Lustre: server umount lustre-MDT0000 complete [ 1019.875836] LustreError: 30036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1019.905449] LustreError: 30036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1023.134792] LustreError: 30023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788469397 with bad export cookie 2874909151781360130 [ 1023.142473] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1023.650083] Lustre: server umount lustre-MDT0001 complete [ 1035.701823] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 1038.393495] Key type lgssc unregistered [ 1038.696661] LNet: 33778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1038.705844] LNetError: 33778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1039.721081] LNet: Removed LNI 192.168.202.135@tcp [ 1040.624373] Key type .llcrypt unregistered [ 1040.626296] Key type ._llcrypt unregistered [ 1059.946814] Key type ._llcrypt registered [ 1059.951639] Key type .llcrypt registered [ 1060.082668] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 1061.402207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1061.738139] alg: No test for adler32 (adler32-zlib) [ 1062.865243] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 1063.202244] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 1064.863288] Key type lgssc registered [ 1066.007541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1075.467744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1075.496986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1077.147467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1083.076402] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1086.432155] LustreError: 35046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1086.442325] LustreError: 35046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1091.556102] LustreError: 35045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1094.066511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1094.383102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1099.582586] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1106.644171] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1112.867455] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1121.399521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1121.995994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1129.408856] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1142.280885] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1146.644512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1160.161451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1160.167703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1160.176932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1162.231273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1162.233331] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1162.253643] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1164.652732] Lustre: server umount lustre-OST0000 complete [ 1171.424396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1171.433443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1171.453270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1171.462206] Lustre: Skipped 1 previous similar message [ 1175.452736] Lustre: server umount lustre-MDT0000 complete [ 1177.568379] LustreError: 35045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1177.581953] LustreError: 35045:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1179.131776] LustreError: 35026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788469553 with bad export cookie 8893992451269918813 [ 1179.136351] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1179.142496] LustreError: 35026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1179.461944] Lustre: server umount lustre-MDT0001 complete [ 1194.187597] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 1197.837277] Key type lgssc unregistered [ 1198.236703] LNet: 38603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1198.247635] LNetError: 38603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1199.274075] LNet: Removed LNI 192.168.202.135@tcp [ 1200.666649] Key type .llcrypt unregistered [ 1200.672167] Key type ._llcrypt unregistered [ 1227.928657] Key type ._llcrypt registered [ 1227.936099] Key type .llcrypt registered [ 1228.035368] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 1229.260781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1229.291438] alg: No test for adler32 (adler32-zlib) [ 1230.356642] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 1230.573603] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 1232.223184] Key type lgssc registered [ 1233.435789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1241.218287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1241.243992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1242.658252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1247.632893] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1252.833807] LustreError: 40017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1252.874665] LustreError: 40017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1257.954074] LustreError: 40016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1258.052445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1258.622716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1265.347984] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1275.654427] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1281.791451] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1289.945608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1290.420402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1297.885246] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1305.392643] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1312.947888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1323.491265] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1323.501764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1323.525622] Lustre: Skipped 1 previous similar message [ 1326.715034] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.35@tcp (stopping) [ 1326.718444] Lustre: Skipped 2 previous similar messages [ 1328.615305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1328.619155] Lustre: Skipped 2 previous similar messages [ 1329.053621] Lustre: server umount lustre-MDT0000 complete [ 1331.788383] LustreError: 40011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1333.013363] LustreError: 39997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788469706 with bad export cookie 119214084695745611 [ 1333.016485] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1333.025842] LustreError: 39997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1333.731756] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1333.742953] Lustre: Skipped 1 previous similar message [ 1333.750577] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1336.915895] LustreError: 40946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1336.959854] LustreError: 40946:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1338.855071] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1338.863167] Lustre: Skipped 1 previous similar message [ 1339.762518] Lustre: server umount lustre-MDT0001 complete [ 1342.944783] Lustre: 42823:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1348.064256] Lustre: 42823:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1353.185364] Lustre: 42821:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1358.304367] Lustre: 42823:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1360.983210] Lustre: server umount lustre-OST0000 complete [ 1380.460289] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 1383.056869] Key type lgssc unregistered [ 1383.235593] LNet: 43900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1383.240311] LNetError: 43900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1383.252177] LNet: Removed LNI 192.168.202.135@tcp [ 1383.832611] Key type .llcrypt unregistered [ 1383.834787] Key type ._llcrypt unregistered [ 1411.248620] Key type ._llcrypt registered [ 1411.252549] Key type .llcrypt registered [ 1411.479932] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 1413.007322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1413.310565] alg: No test for adler32 (adler32-zlib) [ 1414.498372] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 1414.960903] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 1416.671415] Key type lgssc registered [ 1418.215597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1428.040656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1428.061611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1429.987126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1436.539946] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1439.336434] LustreError: 45315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1439.376665] LustreError: 45315:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1444.844384] LustreError: 45314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1449.955581] LustreError: 45315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1452.381527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1453.053395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1458.518656] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1467.036634] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1474.151668] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1483.752162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1484.182157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1494.655664] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1499.811346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1507.902670] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1515.506336] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1515.512704] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1515.536179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1515.552265] Lustre: Skipped 1 previous similar message [ 1516.129448] Lustre: lustre-OST0000: Not available for connect from 192.168.202.35@tcp (stopping) [ 1520.613881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1520.630996] Lustre: Skipped 1 previous similar message [ 1521.818444] Lustre: server umount lustre-OST0000 complete [ 1545.187338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1545.202053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1545.230183] Lustre: Skipped 1 previous similar message [ 1545.248053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1545.255315] Lustre: Skipped 1 previous similar message [ 1546.236234] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1549.730450] Lustre: server umount lustre-MDT0000 complete [ 1552.362187] LustreError: 45310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1552.388148] LustreError: 45310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1554.769161] LustreError: 45295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788469928 with bad export cookie 5932737822255926395 [ 1554.778536] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1554.778978] LustreError: 45295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1569.248572] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1569.667786] Lustre: server umount lustre-MDT0001 complete [ 1580.455469] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 1583.588965] Key type lgssc unregistered [ 1583.822166] LNet: 49029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1583.830784] LNetError: 49029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1583.855700] LNet: Removed LNI 192.168.202.135@tcp [ 1584.732591] Key type .llcrypt unregistered [ 1584.738475] Key type ._llcrypt unregistered [ 1615.304224] Key type ._llcrypt registered [ 1615.311371] Key type .llcrypt registered [ 1615.479057] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 1617.571541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1617.827715] alg: No test for adler32 (adler32-zlib) [ 1618.762507] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 1618.881969] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 1620.543198] Key type lgssc registered [ 1621.989248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1629.891462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1629.902844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1631.236873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1636.742601] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1641.441933] LustreError: 50442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1641.459848] LustreError: 50442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1646.578329] LustreError: 50441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1646.901651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1647.859615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1654.384262] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1664.352029] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1670.070812] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1680.242259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1680.519700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1690.039928] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1699.958255] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1702.057967] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1710.339852] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1713.962420] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1726.431491] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0317h14m58s [ 1728.039632] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1732.215415] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.35@tcp (stopping) [ 1733.096586] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1733.112900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1733.122746] Lustre: Skipped 2 previous similar messages [ 1734.182552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1734.189631] Lustre: Skipped 2 previous similar messages [ 1736.970155] Lustre: server umount lustre-MDT0000 complete [ 1738.322771] LustreError: 51683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1741.060192] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1744.588478] LustreError: 51683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1744.603401] LustreError: 51683:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1745.391028] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1745.425737] Lustre: 53517:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1746.468697] Lustre: 53518:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1748.190287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1748.769825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1748.844784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1749.822772] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1753.315329] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1756.772177] Lustre: DEBUG MARKER: Wait for df (36722) ... [ 1758.349362] Lustre: DEBUG MARKER: done [ 1764.323200] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1764.347395] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1764.376626] Lustre: Skipped 2 previous similar messages [ 1764.423084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1764.437616] Lustre: Skipped 1 previous similar message [ 1764.695628] Lustre: server umount lustre-OST0000 complete [ 1770.472891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1770.494238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1770.498229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1770.515127] Lustre: Skipped 1 previous similar message [ 1775.197366] Lustre: server umount lustre-MDT0000 complete [ 1777.644065] LustreError: 50442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1777.682935] LustreError: 50442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1779.706276] LustreError: 50424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470153 with bad export cookie 10684157180864466860 [ 1779.707261] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1779.722140] LustreError: 50424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1790.397803] Lustre: server umount lustre-MDT0001 complete [ 1809.382596] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 1813.917750] Key type lgssc unregistered [ 1814.361950] LNet: 55395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1814.367395] LNetError: 55395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1814.378785] LNet: Removed LNI 192.168.202.135@tcp [ 1815.208616] Key type .llcrypt unregistered [ 1815.210796] Key type ._llcrypt unregistered [ 1878.614981] Key type ._llcrypt registered [ 1878.621166] Key type .llcrypt registered [ 1878.855761] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 1881.274235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1881.379037] alg: No test for adler32 (adler32-zlib) [ 1882.601729] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 1882.926504] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 1884.655451] Key type lgssc registered [ 1887.021415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1902.373515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1902.422573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1904.265397] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1904.342378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1904.658285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1910.520447] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1919.991422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1920.144375] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1920.196960] Lustre: 57605:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1920.624473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1926.121472] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1936.604627] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1943.543885] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1952.863895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1953.208806] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1953.597309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1960.262826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1963.266143] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1972.343738] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1980.539702] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0317h19m12s [ 1983.111935] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1991.047767] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1993.470764] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1993.886506] LustreError: 59463:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2014.346670] Lustre: lustre-MDT0000: Client 67614150-f7ca-445f-97c7-6ff31eb6231e (at 192.168.202.35@tcp) reconnecting [ 2033.975150] LustreError: 59463:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2036.649846] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2039.749613] Lustre: DEBUG MARKER: done [ 2047.456207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2047.463625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2047.479060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2052.589414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2052.608557] Lustre: Skipped 1 previous similar message [ 2052.978431] Lustre: server umount lustre-OST0000 complete [ 2057.697308] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2057.709537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2057.710072] Lustre: Skipped 2 previous similar messages [ 2057.737994] Lustre: Skipped 2 previous similar messages [ 2063.844062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2063.849773] Lustre: Skipped 3 previous similar messages [ 2070.003301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2070.012204] Lustre: Skipped 1 previous similar message [ 2071.007192] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2071.306536] Lustre: server umount lustre-MDT0000 complete [ 2075.232078] LustreError: 57596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470449 with bad export cookie 14965873220805892544 [ 2075.235051] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2075.248389] LustreError: 57596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2075.879871] Lustre: server umount lustre-MDT0001 complete [ 2087.121644] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 2089.586689] Key type lgssc unregistered [ 2089.879796] LNet: 61941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2089.895706] LNetError: 61941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2089.921143] LNet: Removed LNI 192.168.202.135@tcp [ 2090.781539] Key type .llcrypt unregistered [ 2090.783163] Key type ._llcrypt unregistered [ 2137.264170] Key type ._llcrypt registered [ 2137.271832] Key type .llcrypt registered [ 2137.421554] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 2138.797992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2138.956179] alg: No test for adler32 (adler32-zlib) [ 2140.622441] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 2141.180272] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 2143.024585] Key type lgssc registered [ 2144.591742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2148.419649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2148.436498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2149.592718] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2149.609464] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2149.716240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2150.963616] Lustre: Failing over lustre-MDT0000 [ 2151.368672] Lustre: server umount lustre-MDT0000 complete [ 2158.343352] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 17:22:11 (1788470531) [ 2166.277108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2166.791682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2172.000140] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2181.916928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2182.015143] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2182.058761] Lustre: 64733:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2182.399057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2186.689319] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2193.737428] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2199.374787] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2206.842815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2207.084648] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2207.340257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2214.571133] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2215.155576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2222.227808] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2227.470484] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2235.877992] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2235.891910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2235.898716] Lustre: Skipped 2 previous similar messages [ 2235.904228] Lustre: Skipped 1 previous similar message [ 2240.384575] Lustre: server umount lustre-MDT0000 complete [ 2244.926709] LustreError: 64727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470618 with bad export cookie 17447463436393705887 [ 2244.928039] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2244.940588] LustreError: 64727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2245.370837] Lustre: server umount lustre-MDT0001 complete [ 2249.992800] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2254.309602] Lustre: 67813:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2255.385802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2259.437123] Lustre: 67816:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2262.509429] Lustre: 63659:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788470620/real 1788470620] req@000000006b44b93d x1875347258778880/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788470636 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2262.547185] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2262.557690] Lustre: 67813:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2265.568312] Lustre: 63658:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788470623/real 1788470623] req@00000000c98b3f22 x1875347258779008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788470639 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2266.120389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2266.691748] LustreError: 66584:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2266.717908] LustreError: 66584:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2266.812448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2266.938845] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2267.812730] Lustre: 63657:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788470625/real 1788470625] req@000000002ae4375e x1875347258779136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788470641 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2267.818161] LustreError: 66585:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2267.837977] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2270.881491] Lustre: 63656:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788470628/real 1788470628] req@00000000635fcfb7 x1875347258779392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788470644 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2272.315588] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2272.951114] LustreError: 68510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2272.971675] LustreError: 68510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2276.002125] Lustre: 63658:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788470633/real 1788470633] req@0000000071378e2c x1875347258779648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788470649 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2278.372486] LustreError: 68511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2278.408839] LustreError: 68511:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2281.777383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2282.216341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2287.518352] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2287.601488] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2295.871871] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2302.460740] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2313.191713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2313.194296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2313.199877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2313.220123] Lustre: Skipped 1 previous similar message [ 2313.245489] Lustre: Skipped 2 previous similar messages [ 2315.066396] Lustre: server umount lustre-MDT0000 complete [ 2318.314041] LustreError: 69245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2318.334014] LustreError: 69245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2319.044806] LustreError: 66590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470692 with bad export cookie 17447463436393708190 [ 2319.061366] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2319.062066] LustreError: 66590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2319.528195] Lustre: server umount lustre-MDT0001 complete [ 2326.921063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2328.544719] Lustre: 70578:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2333.670322] Lustre: 70582:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2336.733907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2337.075837] LustreError: 66584:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2337.076440] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2337.107669] Lustre: Skipped 1 previous similar message [ 2337.171919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2337.234722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2339.233309] Lustre: 63656:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788470697/real 1788470697] req@00000000e6a231e1 x1875347258808448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788470713 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2341.779396] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2343.402514] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2352.329140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.264483] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2358.397771] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2366.559679] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2372.270621] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2383.856626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2383.857360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2383.866544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2383.919030] Lustre: Skipped 2 previous similar messages [ 2385.537776] Lustre: server umount lustre-MDT0000 complete [ 2388.961214] LustreError: 72684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2388.982165] LustreError: 72684:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 2389.808679] LustreError: 64727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470763 with bad export cookie 17447463436393709611 [ 2389.811770] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2389.823070] LustreError: 64727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2390.404344] Lustre: server umount lustre-MDT0001 complete [ 2398.527913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2399.206106] Lustre: 73237:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2402.595739] Lustre: DEBUG MARKER: files compared the same [ 2415.331801] Lustre: server umount lustre-OST0000 complete [ 2434.094876] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 2437.239741] Key type lgssc unregistered [ 2437.514723] LNet: 74481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2437.521971] LNetError: 74481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2437.539447] LNet: Removed LNI 192.168.202.135@tcp [ 2438.389478] Key type .llcrypt unregistered [ 2438.391728] Key type ._llcrypt unregistered [ 2463.132952] Key type ._llcrypt registered [ 2463.135406] Key type .llcrypt registered [ 2463.249148] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 2464.637716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2464.701859] alg: No test for adler32 (adler32-zlib) [ 2465.798165] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 2466.073120] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 2467.767264] Key type lgssc registered [ 2468.861815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2475.775762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2475.813322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2477.300419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2481.442457] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug +malloc all [ 2487.777115] LustreError: 75895:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2487.795579] LustreError: 75895:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2489.893102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2490.234283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2494.485995] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug +malloc all [ 2502.276588] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2508.349816] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2515.162099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2515.638688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2522.794359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2523.994812] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug +malloc all [ 2532.546948] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2542.567393] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2542.579613] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2542.596090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2543.585348] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2543.587241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2543.594570] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2543.622198] Lustre: Skipped 1 previous similar message [ 2546.594570] Lustre: server umount lustre-OST0000 complete [ 2553.825498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2553.860780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2553.879312] Lustre: Skipped 1 previous similar message [ 2558.605822] Lustre: server umount lustre-MDT0000 complete [ 2558.943789] LustreError: 75894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.963651] LustreError: 75894:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2562.814603] LustreError: 75876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470936 with bad export cookie 7413032513588719484 [ 2562.815889] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2562.822993] LustreError: 75876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2563.041357] Lustre: server umount lustre-MDT0001 complete [ 2575.454442] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 2578.288158] Key type lgssc unregistered [ 2578.626932] LNet: 79426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2578.645025] LNetError: 79426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2578.679237] LNet: Removed LNI 192.168.202.135@tcp [ 2579.588383] Key type .llcrypt unregistered [ 2579.593801] Key type ._llcrypt unregistered [ 2602.137199] Key type ._llcrypt registered [ 2602.139305] Key type .llcrypt registered [ 2602.288774] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 2603.680343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2603.804066] alg: No test for adler32 (adler32-zlib) [ 2604.958263] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 2605.253458] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 2606.927650] Key type lgssc registered [ 2608.099419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2616.487384] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2616.512124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2617.626192] Lustre: 80825:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2633.119408] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2638.240583] LustreError: 80813:0:(mgc_request.c:317:do_config_log_add()) MGC192.168.202.135@tcp: failed processing log, type 4: rc = -110 [ 2642.913821] Lustre: 80827:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2653.680734] Lustre: 80825:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2658.787324] Lustre: 80827:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2663.904284] Lustre: 80825:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2663.967618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2670.702558] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2674.149269] Lustre: 80825:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2674.178808] Lustre: 80825:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 1 previous similar message [ 2678.154037] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2686.354628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2686.542620] Lustre: *** cfs_fail_loc=706, val=0*** [ 2686.929534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2687.020414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2691.708452] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2692.068190] LustreError: 80842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2692.087852] LustreError: 80842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2697.197899] LustreError: 80843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2698.219446] LustreError: 81904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2698.230782] LustreError: 81904:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2700.495543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2700.796869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2704.490429] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2711.225844] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2717.446890] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2721.250077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2721.255337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2721.269172] Lustre: Skipped 1 previous similar message [ 2726.130554] Lustre: server umount lustre-OST0000 complete [ 2731.488417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2731.488841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2731.500934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2731.500944] Lustre: Skipped 1 previous similar message [ 2731.529319] Lustre: Skipped 1 previous similar message [ 2736.609116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2736.614352] Lustre: Skipped 2 previous similar messages [ 2741.734374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2741.737997] Lustre: Skipped 1 previous similar message [ 2744.287356] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2744.541071] Lustre: server umount lustre-MDT0000 complete [ 2746.852279] LustreError: 81904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2746.883590] LustreError: 81904:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2748.313532] LustreError: 80820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471122 with bad export cookie 8901478607612159354 [ 2748.329793] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2748.330309] LustreError: 80820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2748.738652] Lustre: server umount lustre-MDT0001 complete [ 2758.798025] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 2761.817566] Key type lgssc unregistered [ 2762.273938] LNet: 84444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2762.298820] LNetError: 84444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2762.334788] LNet: Removed LNI 192.168.202.135@tcp [ 2763.348844] Key type .llcrypt unregistered [ 2763.351185] Key type ._llcrypt unregistered [ 2792.530458] Key type ._llcrypt registered [ 2792.536680] Key type .llcrypt registered [ 2792.754040] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 2793.908964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2793.952139] alg: No test for adler32 (adler32-zlib) [ 2795.468899] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 2795.783502] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 2797.537131] Key type lgssc registered [ 2798.966868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2806.808277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2806.857657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2812.602722] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2822.439708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2823.156395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2828.259743] LustreError: 86546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2828.296584] LustreError: 86546:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2828.825666] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2832.354725] LustreError: 86550:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2832.375402] LustreError: 86550:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2837.475523] LustreError: 86546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2837.496367] LustreError: 86546:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2838.250866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2838.663969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2844.144859] LustreError: 86545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2844.162229] LustreError: 86545:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2845.581700] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2852.329508] LustreError: 86550:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2852.356930] LustreError: 86550:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2854.285873] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2854.311427] Lustre: MGS: Client 39c552b2-3e9d-4747-b0f4-31179b117cb0 (at 0@lo) reconnecting [ 2854.341267] LustreError: 86560:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2854.719354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2854.881311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2859.332909] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2875.373626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2875.376968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2875.403063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2878.873952] Lustre: server umount lustre-OST0000 complete [ 2885.600105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2885.601821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2885.607353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2885.621507] Lustre: Skipped 2 previous similar messages [ 2885.642295] Lustre: Skipped 2 previous similar messages [ 2888.916865] Lustre: server umount lustre-MDT0000 complete [ 2890.722718] LustreError: 86549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2893.040409] Lustre: server umount lustre-MDT0001 complete [ 2896.679595] Lustre: server umount MGS complete [ 2908.097607] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 17:34:40 (1788471280) [ 2940.959804] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 2954.305734] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 2991.545518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2995.954051] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3005.162901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3005.353281] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3005.454709] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3005.545358] Lustre: lustre-MDT0001: new disk, initializing [ 3005.629048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3005.659432] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3005.673676] Lustre: 93714:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3005.692050] LustreError: 93714:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3009.673945] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3019.383733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3019.714608] Lustre: lustre-OST0000: new disk, initializing [ 3019.726254] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3019.739488] Lustre: 94478:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3019.816216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3027.631841] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3030.641692] Lustre: 94936:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000005cafa846 x1875347944549504/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788471420 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3036.691585] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3043.345944] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3043.355563] Lustre: MGS: Client 0b935f82-fa28-4da5-9253-fc3ef0da0fbb (at 0@lo) reconnecting [ 3043.382657] LustreError: 93699:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3043.434262] Lustre: 92983:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3043.490579] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3043.577240] Lustre: lustre-MDT0000: new disk, initializing [ 3043.700308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3043.728569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3043.838259] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3044.773356] Lustre: 95548:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000007b21a5c9 x1875347944556416/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788471434 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3049.271519] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3049.842059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3049.850236] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3050.071641] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3075.044905] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3075.048354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3075.071865] Lustre: Skipped 1 previous similar message [ 3075.092586] Lustre: Skipped 1 previous similar message [ 3080.165376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3080.173524] Lustre: Skipped 1 previous similar message [ 3085.285288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3085.294989] Lustre: Skipped 1 previous similar message [ 3089.375381] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3089.542600] Lustre: server umount lustre-OST0000 complete [ 3095.521643] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3095.523360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3095.526520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3095.541648] Lustre: Skipped 1 previous similar message [ 3099.762599] Lustre: server umount lustre-MDT0000 complete [ 3100.641956] LustreError: 93688:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3100.660065] LustreError: 93688:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3110.368157] Lustre: server umount lustre-MDT0001 complete [ 3114.085572] Lustre: server umount MGS complete [ 3128.432122] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 17:38:21 (1788471501) [ 3146.881897] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 3149.752332] Key type lgssc unregistered [ 3150.080325] LNet: 98048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3150.088774] LNetError: 98048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3150.124055] LNet: Removed LNI 192.168.202.135@tcp [ 3151.085786] Key type .llcrypt unregistered [ 3151.090296] Key type ._llcrypt unregistered [ 3170.033511] Key type ._llcrypt registered [ 3170.036744] Key type .llcrypt registered [ 3170.162365] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 3171.290946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3171.354775] alg: No test for adler32 (adler32-zlib) [ 3172.365298] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 3172.632365] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 3174.322745] Key type lgssc registered [ 3175.583978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3183.050166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3183.068696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3184.454749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3189.123324] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3194.852110] LustreError: 99265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3194.859842] LustreError: 99265:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3198.324858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3198.655646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3203.002297] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3209.493084] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3215.322901] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3222.430505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3223.102937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3229.227841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3229.553421] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3237.244444] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3249.644045] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3249.650818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3249.657125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3249.670172] Lustre: Skipped 1 previous similar message [ 3249.685122] LustreError: Skipped 1 previous similar message [ 3254.755168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3254.764722] Lustre: Skipped 2 previous similar messages [ 3259.873704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3259.879379] Lustre: Skipped 1 previous similar message [ 3260.383128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3260.646674] Lustre: server umount lustre-OST0000 complete [ 3264.997778] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3265.003324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3265.019611] Lustre: Skipped 1 previous similar message [ 3270.115786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3270.118802] Lustre: Skipped 2 previous similar messages [ 3270.944679] Lustre: server umount lustre-MDT0000 complete [ 3274.832047] LustreError: 99245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471648 with bad export cookie 15588052286180908723 [ 3274.833878] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3274.847690] LustreError: 99245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3275.183391] Lustre: server umount lustre-MDT0001 complete [ 3287.780101] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 3291.169793] Key type lgssc unregistered [ 3291.719311] LNet: 102799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3291.742095] LNetError: 102799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3291.788616] LNet: Removed LNI 192.168.202.135@tcp [ 3292.811233] Key type .llcrypt unregistered [ 3292.813701] Key type ._llcrypt unregistered [ 3312.142211] Key type ._llcrypt registered [ 3312.144808] Key type .llcrypt registered [ 3312.285981] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 3314.153761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3314.192309] alg: No test for adler32 (adler32-zlib) [ 3315.266258] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 3315.503878] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 3317.167160] Key type lgssc registered [ 3318.374803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3324.767611] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3324.773816] LustreError: 103881:0:(tgt_mount.c:2652:server_fill_super()) cfs_race id 716 sleeping [ 3324.777808] Lustre: Skipped 1 previous similar message [ 3324.789718] LustreError: 103878:0:(tgt_mount.c:2652:server_fill_super()) cfs_fail_race id 716 waking [ 3324.801075] LustreError: 103881:0:(tgt_mount.c:2652:server_fill_super()) cfs_fail_race id 716 awake: rc=4981 [ 3324.809315] LustreError: 103881:0:(tgt_mount.c:2664:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3324.820798] LustreError: 103881:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3324.828268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3326.338592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3334.090104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3334.374420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3338.587211] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3345.272758] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing lsmod [ 3349.373265] LustreError: 105076:0:(tgt_mount.c:2652:server_fill_super()) cfs_race id 716 sleeping [ 3349.421259] LustreError: 105086:0:(tgt_mount.c:2652:server_fill_super()) cfs_fail_race id 716 waking [ 3349.435046] LustreError: 105076:0:(tgt_mount.c:2652:server_fill_super()) cfs_fail_race id 716 awake: rc=4944 [ 3349.451456] LustreError: 105076:0:(tgt_mount.c:2664:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3349.462411] LustreError: 105076:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3349.510143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3349.992233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3352.058110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3355.104378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3355.109058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3355.118036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3357.158518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3357.162644] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3357.223221] Lustre: Skipped 1 previous similar message [ 3359.269095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3359.289473] Lustre: Skipped 4 previous similar messages [ 3359.849297] Lustre: server umount lustre-MDT0000 complete [ 3363.723599] LustreError: 103894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471737 with bad export cookie 2522545728730948133 [ 3363.757124] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3363.796904] Lustre: 105280:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3364.323864] Lustre: 105283:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3364.325367] LustreError: 103913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3364.328509] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3364.356519] LustreError: 103913:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3364.361344] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3365.471381] LustreError: 104349:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000001ea45295 x1875348489788288/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3365.682385] Lustre: server umount lustre-MDT0001 complete [ 3369.444223] Lustre: 105280:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3370.408845] Lustre: server umount lustre-OST0000 complete [ 3379.074335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3379.690088] LustreError: 106154:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3379.786023] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3384.803378] LustreError: 106155:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3384.920665] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3389.928878] LustreError: 106154:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3394.819698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3395.089500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3400.123672] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3407.489971] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3413.502899] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3419.396667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3419.632051] LustreError: 107976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3419.779173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3424.949633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3426.308718] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3434.261600] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3440.615107] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3440.654057] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3440.669207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3440.676372] Lustre: Skipped 1 previous similar message [ 3446.208897] Lustre: server umount lustre-OST0000 complete [ 3450.863693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3450.875080] Lustre: Skipped 1 previous similar message [ 3450.885615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3450.901973] Lustre: Skipped 2 previous similar messages [ 3456.997066] Lustre: server umount lustre-MDT0000 complete [ 3461.091884] LustreError: 106151:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3461.103986] LustreError: 106151:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3461.350769] LustreError: 106134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471835 with bad export cookie 2522545728730949120 [ 3461.371341] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3461.373130] LustreError: 106134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3461.695401] Lustre: server umount lustre-MDT0001 complete [ 3473.801933] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 3477.353372] Key type lgssc unregistered [ 3477.782361] LNet: 109689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3477.798773] LNetError: 109689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3477.828648] LNet: Removed LNI 192.168.202.135@tcp [ 3479.117408] Key type .llcrypt unregistered [ 3479.119529] Key type ._llcrypt unregistered [ 3503.942695] Key type ._llcrypt registered [ 3503.952448] Key type .llcrypt registered [ 3504.073568] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 3505.118521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3505.147101] alg: No test for adler32 (adler32-zlib) [ 3506.472706] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 3506.717624] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 3508.407392] Key type lgssc registered [ 3509.728247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3518.511750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3518.546697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3520.081401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3525.251550] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3529.192694] LustreError: 111111:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3529.228766] LustreError: 111111:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3534.304160] LustreError: 111110:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3534.369521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3534.840270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3539.399154] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3546.942404] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3552.547363] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3559.588542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3560.007986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3564.143889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3566.875116] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3575.303862] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3584.354478] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3587.145863] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3592.785329] Lustre: server umount lustre-OST0000 complete [ 3600.354597] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3600.369615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3600.376938] Lustre: Skipped 1 previous similar message [ 3605.475336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3605.490979] Lustre: Skipped 1 previous similar message [ 3607.660644] Lustre: server umount lustre-MDT0000 complete [ 3610.600250] LustreError: 112118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3610.619675] LustreError: 112118:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3611.427989] LustreError: 111092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471985 with bad export cookie 14480358262964067554 [ 3611.436099] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3611.450313] LustreError: 111092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3622.312555] Lustre: server umount lustre-MDT0001 complete [ 3638.183886] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 3642.535944] Key type lgssc unregistered [ 3642.887117] LNet: 114749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3642.895978] LNetError: 114749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3642.929688] LNet: Removed LNI 192.168.202.135@tcp [ 3643.904197] Key type .llcrypt unregistered [ 3643.908328] Key type ._llcrypt unregistered [ 3666.208163] Key type ._llcrypt registered [ 3666.216459] Key type .llcrypt registered [ 3666.367762] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 3668.095776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3668.219065] alg: No test for adler32 (adler32-zlib) [ 3669.598482] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 3670.094179] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 3671.904555] Key type lgssc registered [ 3673.882305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3682.810984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3682.831115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3684.389717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3690.631378] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3694.564717] LustreError: 115971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3694.576705] LustreError: 115971:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3699.684688] LustreError: 115970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3699.795167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3700.395470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3706.525810] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3714.860663] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3721.078787] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3728.215482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3728.634233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3729.699697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3736.048841] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3744.024125] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3760.617056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3760.631062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3760.660152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3761.633838] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3761.650107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3761.650249] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3761.692348] Lustre: Skipped 1 previous similar message [ 3763.835829] Lustre: server umount lustre-OST0000 complete [ 3771.882372] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3771.889304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3771.893706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3771.898751] Lustre: Skipped 1 previous similar message [ 3774.450876] Lustre: server umount lustre-MDT0000 complete [ 3777.003492] LustreError: 117074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3777.040786] LustreError: 117074:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3778.266364] LustreError: 117815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472152 with bad export cookie 15306511319555785705 [ 3778.273313] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3778.274782] LustreError: 117815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3778.593569] Lustre: server umount lustre-MDT0001 complete [ 3792.454398] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 3796.339074] Key type lgssc unregistered [ 3796.837992] LNet: 119528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3796.843842] LNetError: 119528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3796.881726] LNet: Removed LNI 192.168.202.135@tcp [ 3798.424348] Key type .llcrypt unregistered [ 3798.425929] Key type ._llcrypt unregistered [ 3827.583709] Key type ._llcrypt registered [ 3827.586583] Key type .llcrypt registered [ 3827.686711] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 3828.876139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3828.949913] alg: No test for adler32 (adler32-zlib) [ 3830.643421] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 3831.015489] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 3832.767422] Key type lgssc registered [ 3834.346916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3842.585691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3842.612472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3844.097410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3849.124760] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3853.283864] LustreError: 120958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3853.306690] LustreError: 120958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3858.114582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3858.402472] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3858.630465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3863.510191] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3872.475607] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3878.330943] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3886.075868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3886.327542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3888.390668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3892.372877] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3900.813104] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3911.404077] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3914.926953] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3914.931285] Lustre: Skipped 2 previous similar messages [ 3919.572824] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3919.581144] Lustre: Skipped 2 previous similar messages [ 3921.172354] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3921.211451] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3930.319339] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.35@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3930.329979] Lustre: Skipped 2 previous similar messages [ 3936.747803] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.35@tcp 192.168.0.[2,10]@tcp [ 3943.633098] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.35/24@tcp in log lustre-MDT0000 [ 3943.654423] Lustre: Skipped 2 previous similar messages [ 3946.853772] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.35/24@tcp [ 3946.876833] Lustre: Skipped 1 previous similar message [ 3960.289080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3960.294303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3960.301823] LustreError: Skipped 1 previous similar message [ 3960.302463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3960.324389] Lustre: Skipped 1 previous similar message [ 3960.354030] Lustre: Skipped 1 previous similar message [ 3962.027806] Lustre: server umount lustre-OST0000 complete [ 3966.432474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3966.441723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3966.455498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3970.528660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3970.535661] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3972.498277] Lustre: server umount lustre-MDT0000 complete [ 3975.656663] LustreError: 120953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3975.683115] LustreError: 120953:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3977.108218] LustreError: 124590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472351 with bad export cookie 7006276222408841102 [ 3977.115338] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3977.126548] LustreError: 124590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3977.428146] Lustre: server umount lustre-MDT0001 complete [ 3991.707334] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 3995.263240] Key type lgssc unregistered [ 3995.596779] LNet: 125748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3995.601623] LNetError: 125748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3995.611784] LNet: Removed LNI 192.168.202.135@tcp [ 3996.637193] Key type .llcrypt unregistered [ 3996.638876] Key type ._llcrypt unregistered [ 4028.460455] Key type ._llcrypt registered [ 4028.464805] Key type .llcrypt registered [ 4028.602257] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 4029.867761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4029.899350] alg: No test for adler32 (adler32-zlib) [ 4030.970455] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 4031.193891] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 4032.871179] Key type lgssc registered [ 4033.948310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4044.395903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4044.433766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4046.230930] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4046.261338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4046.274016] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.35/24@tcp [ 4052.399801] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 4056.546636] LustreError: 127307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4056.569930] LustreError: 127307:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4061.667281] LustreError: 127308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4062.473903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4062.797720] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4062.855905] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.35/24@tcp [ 4062.934754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4068.023264] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 4076.554439] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4082.188925] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4089.557460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4089.984076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4092.056226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4096.892509] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 4105.152606] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4123.104828] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4123.112120] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4123.119030] LustreError: Skipped 1 previous similar message [ 4123.127851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4123.157513] Lustre: Skipped 1 previous similar message [ 4124.998204] Lustre: server umount lustre-OST0000 complete [ 4129.760464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4129.771870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4129.793976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4129.801087] Lustre: Skipped 1 previous similar message [ 4133.347441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4133.368504] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4135.869229] Lustre: server umount lustre-MDT0000 complete [ 4138.464496] LustreError: 129816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4138.485603] LustreError: 129816:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4139.636144] LustreError: 127287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472513 with bad export cookie 3751313352040748630 [ 4139.651051] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4139.657228] LustreError: 127287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4140.000956] Lustre: server umount lustre-MDT0001 complete [ 4152.573902] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 4155.995374] Key type lgssc unregistered [ 4156.543876] LNet: 130897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4156.559735] LNetError: 130897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4157.608703] LNet: Removed LNI 192.168.202.135@tcp [ 4159.155939] Key type .llcrypt unregistered [ 4159.161486] Key type ._llcrypt unregistered [ 4193.366337] Key type ._llcrypt registered [ 4193.369279] Key type .llcrypt registered [ 4193.481967] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 4210.250834] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 4211.599494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4211.669986] alg: No test for adler32 (adler32-zlib) [ 4212.726311] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 4213.123977] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 4214.887355] Key type lgssc registered [ 4216.186458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4254.822946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4254.851585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4256.171989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4256.237436] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4256.388895] Lustre: lustre-MDT0000: new disk, initializing [ 4256.532740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4256.590644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4262.013243] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 4274.237558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4274.397761] Lustre: 134295:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4274.522464] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4274.532836] Lustre: Skipped 1 previous similar message [ 4274.645239] Lustre: lustre-MDT0001: new disk, initializing [ 4274.740713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4274.780794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4274.794346] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4281.472228] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 4293.726724] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4300.688473] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4308.535118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4308.769472] Lustre: lustre-OST0000: new disk, initializing [ 4308.772760] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4308.777514] Lustre: 136263:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4308.854396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4316.355342] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 4318.318500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4318.341416] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4318.397488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4327.012867] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4333.499488] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4333.767424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4337.596477] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4337.957146] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4344.291381] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4344.298930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4344.309754] LustreError: Skipped 1 previous similar message [ 4344.313640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4344.346220] Lustre: Skipped 1 previous similar message [ 4344.380327] Lustre: Skipped 1 previous similar message [ 4346.650373] Lustre: server umount lustre-OST0000 complete [ 4351.456481] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4351.465718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4351.506628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4354.541511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4354.546707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4354.581626] Lustre: Skipped 1 previous similar message [ 4356.809710] Lustre: server umount lustre-MDT0000 complete [ 4361.247453] LustreError: 134289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472735 with bad export cookie 5120247764693285330 [ 4361.261484] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4361.268901] LustreError: 134289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4361.872916] Lustre: server umount lustre-MDT0001 complete [ 4370.910672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4371.467366] LustreError: 138482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4371.514810] LustreError: 138482:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4371.591882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4376.805674] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 4377.062504] LustreError: 138483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4381.154779] LustreError: 138482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4385.862702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4386.272299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4390.536673] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 4397.507711] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4404.074251] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4411.325882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4411.691836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4418.417123] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 4427.401462] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4432.438260] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4432.663104] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4438.464537] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4438.730473] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4447.942958] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4448.174717] Lustre: lustre-OST0001: new disk, initializing [ 4448.178938] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4448.189932] Lustre: 141859:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4448.319724] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4455.801579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4455.820530] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4455.971717] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4456.087124] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 4466.621578] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4466.908733] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4472.027120] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4472.312726] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4480.572541] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff95f146caa000.ost_server_uuid 50 [ 4482.335234] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff95f146caa000.ost_server_uuid in IDLE state after 0 sec [ 4496.865543] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4496.872065] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4496.893763] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4496.900828] Lustre: Skipped 1 previous similar message [ 4496.921051] Lustre: Skipped 1 previous similar message [ 4499.112307] Lustre: server umount lustre-OST0001 complete [ 4502.019452] LustreError: 140310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4502.046170] LustreError: 140310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4504.040710] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4504.060406] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4504.101100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4507.110692] LustreError: 140311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4507.126945] LustreError: 140311:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4512.229598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4512.243177] Lustre: Skipped 4 previous similar messages [ 4517.343595] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4517.374139] LustreError: 140310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4517.412500] LustreError: 140310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4517.976784] Lustre: server umount lustre-OST0000 complete [ 4522.465141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4522.475493] Lustre: Skipped 1 previous similar message [ 4528.459342] Lustre: server umount lustre-MDT0000 complete [ 4534.442199] LustreError: 138463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472908 with bad export cookie 5120247764693286226 [ 4534.456344] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4534.472581] LustreError: 138463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4534.511197] Lustre: 143846:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4535.202783] Lustre: server umount lustre-MDT0001 complete [ 4555.737889] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 4558.991803] Key type lgssc unregistered [ 4559.345938] LNet: 144879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4559.349978] LNetError: 144879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4560.422554] LNet: Removed LNI 192.168.202.135@tcp [ 4561.254512] Key type .llcrypt unregistered [ 4561.257261] Key type ._llcrypt unregistered [ 4603.263921] Key type ._llcrypt registered [ 4603.266954] Key type .llcrypt registered [ 4603.405333] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 4619.999401] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 4621.612274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4621.629907] alg: No test for adler32 (adler32-zlib) [ 4622.974277] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 4623.264396] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 4624.951365] Key type lgssc registered [ 4626.719195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4663.130832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4663.158308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4664.459551] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4664.493334] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4664.605367] Lustre: lustre-MDT0000: new disk, initializing [ 4664.688060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4664.718867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4669.983603] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 4682.649630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4682.777897] Lustre: 149032:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4682.869910] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4682.875096] Lustre: Skipped 1 previous similar message [ 4682.982510] Lustre: lustre-MDT0001: new disk, initializing [ 4683.136917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4683.175374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4683.200103] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4687.744308] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 4697.919757] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4704.185868] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4710.263541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4710.507564] Lustre: lustre-OST0000: new disk, initializing [ 4710.511484] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4710.517781] Lustre: 151001:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4710.606663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4716.878779] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 4717.668872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4717.687586] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4717.859644] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4727.242915] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4731.605210] Lustre: Failing over lustre-OST0000 [ 4733.414087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4733.419305] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4733.430686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4733.431269] Lustre: Skipped 1 previous similar message [ 4733.442372] LustreError: Skipped 1 previous similar message [ 4733.968541] Lustre: server umount lustre-OST0000 complete [ 4752.536184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4752.933023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4752.962077] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4753.958764] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4760.415887] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 4760.712341] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4760.712619] Lustre: lustre-OST0000: Recovery over after 0:07, of 3 clients 3 recovered and 0 were evicted. [ 4764.913603] Lustre: Failing over lustre-MDT0000 [ 4765.152189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4765.160777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4765.209665] Lustre: server umount lustre-MDT0000 complete [ 4769.250614] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4769.276024] Lustre: Skipped 1 previous similar message [ 4770.886487] LustreError: 149037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4770.911572] LustreError: 149037:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4774.380317] LustreError: 149039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4774.392462] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4774.408600] LustreError: 149039:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4774.428448] Lustre: 152922:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4776.010747] LustreError: 149037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4779.017996] LustreError: 152536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4779.491665] Lustre: 152920:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4784.205271] LustreError: 149038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4784.224170] LustreError: 149038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4784.591236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4784.613660] Lustre: 152920:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4785.253548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4785.316936] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4789.329984] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4790.256275] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4790.271041] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4790.312503] Lustre: Skipped 1 previous similar message [ 4790.372858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4791.087386] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 4800.482399] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4800.505542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4800.522981] Lustre: Skipped 2 previous similar messages [ 4805.601702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4805.607392] Lustre: Skipped 1 previous similar message [ 4806.704885] Lustre: server umount lustre-OST0000 complete [ 4815.841658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4815.847618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4815.876518] Lustre: Skipped 2 previous similar messages [ 4815.916707] Lustre: Skipped 1 previous similar message [ 4818.096653] Lustre: server umount lustre-MDT0000 complete [ 4820.960596] LustreError: 149042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4821.009700] LustreError: 149042:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4824.926402] LustreError: 149024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788473198 with bad export cookie 17674463493967820571 [ 4824.934347] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4824.944610] LustreError: 149024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4826.567209] Lustre: server umount lustre-MDT0001 complete [ 4841.587717] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 4844.890624] Key type lgssc unregistered [ 4845.275724] LNet: 154664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4845.287269] LNetError: 154664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4845.319898] LNet: Removed LNI 192.168.202.135@tcp [ 4846.477564] Key type .llcrypt unregistered [ 4846.484280] Key type ._llcrypt unregistered [ 4886.337943] Key type ._llcrypt registered [ 4886.339832] Key type .llcrypt registered [ 4886.488490] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 4888.365614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4888.431052] alg: No test for adler32 (adler32-zlib) [ 4889.578653] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 4889.821771] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 4891.575180] Key type lgssc registered [ 4892.717740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4902.500009] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4902.520058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4904.128864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4909.392765] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 4913.638911] LustreError: 156183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4913.666756] LustreError: 156183:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4918.755299] LustreError: 156184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4920.563688] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4923.882548] LustreError: 156183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4925.072429] LustreError: 156212:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4925.086449] LustreError: 156212:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4925.100270] LustreError: 156212:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 4925.661127] Lustre: server umount lustre-MDT0000 complete [ 4944.223503] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 4956.301519] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 4998.968974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4999.464472] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4999.526433] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4999.678534] Lustre: lustre-MDT0000: new disk, initializing [ 4999.791458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4999.816215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5007.086570] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 5020.631365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5020.779778] Lustre: 159519:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5020.875865] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5020.885599] Lustre: Skipped 1 previous similar message [ 5020.984556] Lustre: lustre-MDT0001: new disk, initializing [ 5021.124377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5021.177095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5021.193901] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5025.949490] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 5037.440873] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5044.434311] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5053.003175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5053.470395] Lustre: lustre-OST0000: new disk, initializing [ 5053.480204] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5053.485743] Lustre: 161489:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5053.613976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5060.551116] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 5061.807143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5061.826849] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5062.050518] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5071.191261] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5076.403426] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5078.750587] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 18:10:51 (1788473451) [ 5302.752301] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5302.752938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5302.753410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5302.771690] Lustre: Skipped 2 previous similar messages [ 5302.795147] Lustre: Skipped 2 previous similar messages [ 5304.429878] Lustre: server umount lustre-MDT0000 complete [ 5307.872846] LustreError: 160330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5307.894802] LustreError: 160330:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5308.213857] LustreError: 159512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788473682 with bad export cookie 14396485682616845628 [ 5308.218369] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5308.224091] LustreError: 159512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5312.991880] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5312.993151] LustreError: 163259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5313.008067] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5314.017502] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5314.722964] Lustre: server umount lustre-MDT0001 complete [ 5318.121900] Lustre: 163584:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 5323.234616] Lustre: 163586:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 5324.846878] Lustre: server umount lustre-OST0000 complete [ 5337.082602] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 5348.086172] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 5389.819204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5390.061807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5390.101816] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5390.169862] Lustre: lustre-MDT0000: new disk, initializing [ 5390.220274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5390.234086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5396.204863] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 5410.837257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5411.082683] Lustre: 166152:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5411.260355] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5411.273304] Lustre: Skipped 1 previous similar message [ 5411.403479] Lustre: lustre-MDT0001: new disk, initializing [ 5411.617475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5411.686071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5411.700464] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5417.894393] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 5429.318435] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5436.073564] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5443.365811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5443.758620] Lustre: lustre-OST0000: new disk, initializing [ 5443.765710] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5443.782818] Lustre: 168123:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5443.929436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5451.874829] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 5452.374407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5452.382865] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5452.500907] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5461.445894] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5466.613860] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5466.955598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5472.405215] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5472.654919] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5478.369879] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5478.370339] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5478.386485] LustreError: Skipped 1 previous similar message [ 5478.412324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5478.441085] Lustre: Skipped 1 previous similar message [ 5481.530292] Lustre: server umount lustre-OST0000 complete [ 5488.609417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5488.618453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5488.623102] Lustre: Skipped 1 previous similar message [ 5488.626304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5488.666254] Lustre: Skipped 1 previous similar message [ 5493.237355] Lustre: server umount lustre-MDT0000 complete [ 5493.728944] LustreError: 166164:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5493.781629] LustreError: 166164:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5497.676606] LustreError: 166147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788473871 with bad export cookie 14396485682617004479 [ 5497.702918] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5497.704685] LustreError: 166147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5498.134059] Lustre: server umount lustre-MDT0001 complete [ 5508.084071] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:18:00 (1788473880) [ 5527.968370] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 5541.853923] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 5591.683045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5592.164206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5592.221350] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5592.326323] Lustre: lustre-MDT0000: new disk, initializing [ 5592.414435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5592.431535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5598.333533] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 5608.572193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5608.664470] Lustre: 172882:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5608.884516] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5612.976776] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 5623.060704] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5629.295943] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5638.131781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5638.373643] Lustre: lustre-OST0000: new disk, initializing [ 5638.377550] Lustre: Skipped 1 previous similar message [ 5638.382260] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5638.392480] Lustre: Skipped 2 previous similar messages [ 5638.399125] Lustre: 174857:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5645.827995] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 5646.066368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5646.104484] Lustre: Skipped 1 previous similar message [ 5646.121614] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5646.311550] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5657.795438] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5671.916078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5671.919059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5671.920272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5671.970784] Lustre: Skipped 2 previous similar messages [ 5673.987027] Lustre: server umount lustre-OST0000 complete [ 5680.608478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5680.616978] LustreError: Skipped 1 previous similar message [ 5680.625776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5691.363130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5691.375814] Lustre: Skipped 7 previous similar messages [ 5692.383190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5692.736568] Lustre: server umount lustre-MDT0000 complete [ 5696.216624] LustreError: 172873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788474070 with bad export cookie 14396485682617005382 [ 5696.218400] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5696.226522] LustreError: 172873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5696.542641] Lustre: server umount lustre-MDT0001 complete [ 5705.482979] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:21:18 (1788474078) [ 5724.879657] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 5735.516068] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 5777.452258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5777.893908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5777.936910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5778.041530] Lustre: lustre-MDT0000: new disk, initializing [ 5778.131185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5778.136583] Lustre: Skipped 2 previous similar messages [ 5778.157630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5784.308791] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 5798.071792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5798.158111] Lustre: 179184:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5798.451203] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5804.631694] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 5814.575560] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5821.362665] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5828.533986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5829.099306] Lustre: 181161:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5836.509731] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5836.684166] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5837.593040] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 5848.981084] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5857.251654] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5857.261356] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5857.263544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5857.276911] LustreError: Skipped 1 previous similar message [ 5857.338124] Lustre: Skipped 2 previous similar messages [ 5863.386828] Lustre: server umount lustre-OST0000 complete [ 5882.335158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5882.585065] Lustre: server umount lustre-MDT0000 complete [ 5882.848601] LustreError: 179190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5882.858836] LustreError: 179190:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5886.138195] LustreError: 179175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788474260 with bad export cookie 14396485682617006474 [ 5886.141413] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5886.153146] LustreError: 179175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5886.758346] Lustre: server umount lustre-MDT0001 complete [ 5899.520671] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 5902.264363] Key type lgssc unregistered [ 5902.503880] LNet: 183019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5902.508091] LNetError: 183019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5902.529149] LNet: Removed LNI 192.168.202.135@tcp [ 5903.460316] Key type .llcrypt unregistered [ 5903.462664] Key type ._llcrypt unregistered [ 5930.565195] Key type ._llcrypt registered [ 5930.569188] Key type .llcrypt registered [ 5930.817319] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 5932.443064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5932.548746] alg: No test for adler32 (adler32-zlib) [ 5933.719582] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 5934.100471] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 5935.857125] Key type lgssc registered [ 5937.099955] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5944.653746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5944.677479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5946.387946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5951.902267] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 5956.590158] LustreError: 184432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5956.637560] LustreError: 184432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5961.697071] LustreError: 184431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5962.533612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5963.556541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5969.628506] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 5978.786786] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5986.378995] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5992.607770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5993.034200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5999.877965] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 6008.721512] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6016.641223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6039.523039] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6039.530173] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6039.541221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6042.596566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6042.600343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6044.788538] Lustre: server umount lustre-OST0000 complete [ 6050.793057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6050.808676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6050.856985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6050.880319] Lustre: Skipped 1 previous similar message [ 6052.835301] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6052.865326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6052.882859] Lustre: Skipped 1 previous similar message [ 6056.830665] Lustre: server umount lustre-MDT0000 complete [ 6057.958299] LustreError: 184426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6057.993270] LustreError: 184426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6062.398059] LustreError: 184413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788474436 with bad export cookie 10155279473715390986 [ 6062.406357] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6062.410584] LustreError: 184413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6062.425730] Lustre: 187419:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 6062.882231] Lustre: server umount lustre-MDT0001 complete [ 6079.046697] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 6082.476621] Key type lgssc unregistered [ 6082.868412] LNet: 188000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6082.896625] LNetError: 188000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6082.920023] LNet: Removed LNI 192.168.202.135@tcp [ 6084.215370] Key type .llcrypt unregistered [ 6084.217388] Key type ._llcrypt unregistered [ 6110.734148] Key type ._llcrypt registered [ 6110.736029] Key type .llcrypt registered [ 6110.932372] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 6112.379557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6112.475146] alg: No test for adler32 (adler32-zlib) [ 6113.641246] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 6113.915462] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 6115.568500] Key type lgssc registered [ 6116.677797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6123.455281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6123.478129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6125.063138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6130.346822] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 6134.256257] LustreError: 189412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6134.276066] LustreError: 189412:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6138.778529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6139.234272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6143.849309] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 6153.734716] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6160.238755] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6166.698837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6167.055333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6169.151912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6173.953705] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 6180.987504] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6189.995634] Lustre: server umount lustre-OST0000 complete [ 6197.221779] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff95f158449800.ost_server_uuid 50 [ 6211.039331] Lustre: 188723:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788474568/real 1788474568] req@00000000ee85de74 x1875351423707520/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788474584 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6211.041809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6211.062743] Lustre: 188723:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6211.086682] Lustre: Skipped 1 previous similar message [ 6212.642506] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6215.136047] Lustre: 188723:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788474573/real 1788474573] req@00000000fe6fd0e6 x1875351423709056/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788474589 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6215.164663] Lustre: 188723:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6220.269618] Lustre: 188723:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788474578/real 1788474578] req@00000000a2d97696 x1875351423710208/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788474594 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6220.306459] Lustre: 188723:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6225.375180] Lustre: 188724:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788474583/real 1788474583] req@00000000e8f41dd9 x1875351423711744/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788474599 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6225.418576] Lustre: 188724:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 6226.400537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6226.408994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6226.422369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6230.498157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6230.500224] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6232.097759] Lustre: server umount lustre-MDT0000 complete [ 6235.616726] LustreError: 190602:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6235.626584] LustreError: 190602:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6236.493673] LustreError: 189392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788474610 with bad export cookie 5955395880981094200 [ 6236.505085] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6236.519512] LustreError: 189392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6237.015721] Lustre: server umount lustre-MDT0001 complete [ 6246.362476] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 18:30:19 (1788474619) [ 6253.696935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6254.292158] LustreError: 193348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6254.366792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6259.333721] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 6259.681827] LustreError: 193349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6264.801458] LustreError: 193348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6267.969152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6268.334713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6273.194134] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 6280.117517] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6285.435937] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6292.642927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6293.148922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6299.320897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6300.157776] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 6310.173066] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6318.887362] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6319.123501] Lustre: lustre-OST0001: new disk, initializing [ 6319.126984] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6319.132766] Lustre: 196190:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6319.244582] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6325.556262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6325.568838] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6325.668502] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6327.172252] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 6338.779430] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6344.821807] Lustre: lustre-OST0000: Not available for connect from 192.168.202.35@tcp (stopping) [ 6344.834887] Lustre: Skipped 1 previous similar message [ 6345.696583] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6345.706682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6347.753225] Lustre: server umount lustre-OST0000 complete [ 6349.898900] LustreError: 195169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6355.205470] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6355.442419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6360.083529] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6360.313450] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6366.693930] LustreError: 195693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6366.723406] LustreError: 195693:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 6376.928319] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6376.929374] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6376.940340] LustreError: Skipped 2 previous similar messages [ 6376.955714] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6376.974996] Lustre: Skipped 2 previous similar messages [ 6376.999688] Lustre: Skipped 5 previous similar messages [ 6380.404612] Lustre: server umount lustre-OST0001 complete [ 6386.146332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6386.167276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6386.199525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6392.471200] Lustre: server umount lustre-MDT0000 complete [ 6396.528099] LustreError: 193330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788474770 with bad export cookie 5955395880981095299 [ 6396.532871] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6396.549616] LustreError: 193330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6397.061159] Lustre: server umount lustre-MDT0001 complete [ 6421.572040] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 18:33:14 (1788474794) [ 6428.687654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6429.031696] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6429.060431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6429.213448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6433.519411] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 6441.668900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6441.768658] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6441.806986] Lustre: 199635:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6446.978299] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 6454.230958] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6459.935854] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6467.252684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6467.487274] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6474.878326] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 6477.968352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6482.610732] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6488.769122] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6488.864922] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6495.071912] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6495.521816] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 6503.603657] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6508.517525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6508.528069] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6508.538873] Lustre: Skipped 1 previous similar message [ 6508.543208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6508.547519] Lustre: Skipped 2 previous similar messages [ 6512.545556] Lustre: server umount lustre-OST0000 complete [ 6514.781582] LustreError: 201485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6514.803047] LustreError: 201485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6531.045428] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6531.047734] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6531.054874] Lustre: Skipped 4 previous similar messages [ 6531.066026] LustreError: Skipped 1 previous similar message [ 6535.650125] Lustre: server umount lustre-OST0001 complete [ 6541.281484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6541.294749] Lustre: Skipped 4 previous similar messages [ 6547.224808] Lustre: server umount lustre-MDT0000 complete [ 6551.003529] LustreError: 199626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788474924 with bad export cookie 5955395880981096762 [ 6551.011171] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6551.023277] LustreError: 199626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6551.531435] Lustre: server umount lustre-MDT0001 complete [ 6581.049824] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 18:35:53 (1788474953) [ 6597.659406] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 6606.488734] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 6643.559377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6643.805248] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6643.830341] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6643.915631] Lustre: lustre-MDT0000: new disk, initializing [ 6644.029689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6644.042980] Lustre: Skipped 3 previous similar messages [ 6644.077688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6648.677886] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 6659.155233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6659.278921] Lustre: 207730:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6659.400364] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6659.405821] Lustre: Skipped 1 previous similar message [ 6659.502413] Lustre: lustre-MDT0001: new disk, initializing [ 6659.731351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6659.754871] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6664.467940] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 6673.443660] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6678.648278] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6684.591103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6684.862234] Lustre: lustre-OST0000: new disk, initializing [ 6684.864813] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6684.871914] Lustre: 209699:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6690.925658] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 6692.009924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6692.031126] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6692.209154] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6700.217790] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6704.722269] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6704.958815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6708.986927] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6709.183164] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6712.803605] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6712.805751] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6712.833335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6712.850508] Lustre: Skipped 5 previous similar messages [ 6717.419725] Lustre: server umount lustre-OST0000 complete [ 6735.330394] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6735.695954] Lustre: server umount lustre-MDT0000 complete [ 6737.377062] LustreError: 207741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6737.394063] LustreError: 207741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6739.245060] LustreError: 207721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788475113 with bad export cookie 5955395880981098582 [ 6739.256172] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6739.262917] LustreError: 207721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6739.585668] Lustre: server umount lustre-MDT0001 complete [ 6747.464875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6751.731810] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 6759.805812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6764.319759] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 6772.056626] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6778.444589] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6785.388186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6785.747725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6785.757534] Lustre: Skipped 4 previous similar messages [ 6793.864258] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 6802.716356] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6807.369292] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6807.662032] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6813.288482] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6813.537736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6818.784106] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6818.795994] LustreError: Skipped 1 previous similar message [ 6818.806708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6818.814943] Lustre: Skipped 8 previous similar messages [ 6821.852504] Lustre: server umount lustre-OST0000 complete [ 6829.340977] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6834.976045] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6865.015249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6869.742175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6869.775383] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6873.956407] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 6884.143700] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6884.193324] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6884.874664] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6887.141756] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6894.562186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6894.597266] Lustre: Skipped 5 previous similar messages [ 6898.711804] Lustre: server umount lustre-OST0000 complete [ 6903.776355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6903.783606] LustreError: Skipped 1 previous similar message [ 6908.936717] Lustre: server umount lustre-MDT0000 complete [ 6909.920577] LustreError: 211919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6909.943995] LustreError: 211919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6912.674527] LustreError: 211898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788475286 with bad export cookie 5955395880981099485 [ 6912.679396] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6912.686043] LustreError: 211898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6912.999800] Lustre: server umount lustre-MDT0001 complete [ 6922.916403] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 18:41:35 (1788475295) [ 6930.353486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6936.439441] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 6946.523754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6951.256739] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 6960.361204] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6965.904345] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6973.674661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6981.048293] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 6990.022713] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6996.786860] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6997.210473] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7005.473921] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7005.764499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7015.194056] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7015.414937] Lustre: lustre-OST0001: new disk, initializing [ 7015.417773] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7015.422527] Lustre: 221254:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7023.176221] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7024.941691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7024.952151] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7025.015709] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7034.782032] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7040.566423] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7040.846670] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7046.770730] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7047.038880] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7050.721471] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7050.765035] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7050.793076] Lustre: Skipped 8 previous similar messages [ 7055.968401] Lustre: server umount lustre-OST0001 complete [ 7063.062480] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7063.405535] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7068.537242] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7068.747178] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7100.752180] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7101.257592] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7101.264485] Lustre: Skipped 5 previous similar messages [ 7106.543562] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7106.587471] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7106.599061] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7108.718444] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7116.905665] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7118.567405] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7127.104622] Lustre: server umount lustre-OST0001 complete [ 7157.217025] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7157.697982] Lustre: server umount lustre-OST0000 complete [ 7161.825965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7161.843938] Lustre: Skipped 9 previous similar messages [ 7167.961892] Lustre: server umount lustre-MDT0000 complete [ 7168.008834] LustreError: 218611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7168.041452] LustreError: 218611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 42 previous similar messages [ 7173.142394] LustreError: 217864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788475547 with bad export cookie 5955395880981100493 [ 7173.144338] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7173.163356] LustreError: 217864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7179.705055] Lustre: server umount lustre-MDT0001 complete [ 7205.175771] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 18:46:17 (1788475577) [ 7213.374507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7213.709446] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7213.756034] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7220.470394] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7231.048732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7231.169330] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7231.210672] Lustre: 226502:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7238.205900] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7247.112018] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7254.678488] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7262.090585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7262.304246] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7270.014184] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7278.770821] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7287.107731] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7287.216772] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7294.058349] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7302.014595] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7303.850434] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7309.221958] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7309.555363] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7316.173861] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7316.504302] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7322.826930] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff95f14464d800.ost_server_uuid 50 [ 7324.297529] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff95f14464d800.ost_server_uuid in IDLE state after 0 sec [ 7325.177316] Lustre: Permanently deactivating lustre-OST0001 [ 7325.190487] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7328.734253] Lustre: Permanently reactivating lustre-OST0001 [ 7335.905257] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7335.917463] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7335.929241] LustreError: Skipped 6 previous similar messages [ 7335.935310] Lustre: Skipped 12 previous similar messages [ 7336.230635] Lustre: server umount lustre-OST0001 complete [ 7355.368777] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7355.580871] Lustre: server umount lustre-OST0000 complete [ 7367.257802] Lustre: server umount lustre-MDT0000 complete [ 7371.486783] LustreError: 229379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788475745 with bad export cookie 5955395880981101851 [ 7371.488222] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7371.508555] LustreError: 229379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7371.919833] Lustre: server umount lustre-MDT0001 complete [ 7396.512597] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 18:49:29 (1788475769) [ 7403.729724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7404.065148] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7404.111521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7404.118968] Lustre: Skipped 5 previous similar messages [ 7409.141339] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7419.123213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7419.227273] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7419.250450] Lustre: 233204:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7424.520259] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7431.973946] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7439.054878] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7446.290406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7446.471383] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7446.505416] Lustre: Permanently deactivating lustre-OST0000 [ 7446.517803] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7455.345899] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7456.880312] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7456.906803] LustreError: 235495:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -5 [ 7468.499695] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7477.858190] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7477.979150] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7485.376378] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7486.392959] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7494.532172] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7496.508240] Lustre: Permanently reactivating lustre-OST0000 [ 7496.520882] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7496.525029] Lustre: Skipped 2 previous similar messages [ 7503.091395] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7503.092839] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 46 seconds. I think it's dead, and I am evicting it. exp 00000000404f8600, cur 1788475876 deadline 1788475860 last 1788475830 [ 7503.098130] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7503.099780] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7503.134626] Lustre: Skipped 1 previous similar message [ 7503.146421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7523.183613] Lustre: server umount lustre-OST0001 complete [ 7543.263139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7543.534269] Lustre: server umount lustre-OST0000 complete [ 7555.267529] Lustre: server umount lustre-MDT0000 complete [ 7558.062500] LustreError: 235074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788475931 with bad export cookie 5955395880981103615 [ 7558.070472] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7558.907811] Lustre: server umount lustre-MDT0001 complete [ 7572.234701] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 7575.599409] Key type lgssc unregistered [ 7576.141659] LNet: 238089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7576.147359] LNetError: 238089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7576.165146] LNet: Removed LNI 192.168.202.135@tcp [ 7577.417160] Key type .llcrypt unregistered [ 7577.419046] Key type ._llcrypt unregistered [ 7602.895316] Key type ._llcrypt registered [ 7602.898755] Key type .llcrypt registered [ 7603.027480] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 7604.827898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7605.046586] alg: No test for adler32 (adler32-zlib) [ 7606.259813] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 7606.507354] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 7608.175364] Key type lgssc registered [ 7609.979359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7622.414995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7622.431578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7623.865710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7628.963486] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7633.380267] LustreError: 239554:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7633.421670] LustreError: 239554:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7637.503553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7637.614442] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7637.634663] Lustre: 239542:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7637.647947] Lustre: Permanently deactivating lustre-MDT0001 [ 7637.661569] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7637.896074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7641.116334] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7641.131204] LustreError: 239585:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7641.150769] LustreError: 239585:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7641.162602] LustreError: 239585:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 1, failed: rc = -108 [ 7642.130736] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7649.130510] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7654.051839] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7660.535206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7660.899694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7662.969274] LustreError: 241396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7662.989091] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7667.519451] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7668.138697] LustreError: 241537:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7668.148551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7668.153426] LustreError: 241537:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7673.321425] LustreError: 241396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7673.344949] LustreError: 241396:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7673.736630] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7678.438831] LustreError: 241537:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7678.453975] LustreError: 241537:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7680.529191] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7680.847780] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7686.142409] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7686.164259] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7686.458988] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7693.996168] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7697.429887] Lustre: Permanently reactivating lustre-MDT0001 [ 7697.455539] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7697.459673] Lustre: Skipped 2 previous similar messages [ 7702.136434] Lustre: lustre-MDT0001: Client 04fa436b-ae50-4a7a-b87e-b8b98dff3afc (at 192.168.202.35@tcp) reconnecting [ 7709.645789] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7712.010225] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7719.326919] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7719.586342] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7724.772838] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7725.064298] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7727.556409] Lustre: Permanently deactivating lustre-MDT0001 [ 7727.582427] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7727.590854] Lustre: Skipped 2 previous similar messages [ 7734.544565] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7740.408587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7740.420242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7740.442215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7742.268635] Lustre: server umount lustre-MDT0000 complete [ 7742.448333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7742.450533] LustreError: 239555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7742.468469] Lustre: Skipped 1 previous similar message [ 7742.509466] LustreError: 239555:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7746.053291] LustreError: 243102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788476119 with bad export cookie 18147148930885493186 [ 7746.054649] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7746.066630] LustreError: 243102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7746.489422] Lustre: server umount lustre-MDT0001 complete [ 7752.296741] Lustre: server umount lustre-OST0000 complete [ 7755.695235] Lustre: 244531:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7760.867089] Lustre: 244533:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7762.913932] Lustre: 244531:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7765.232596] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 18:55:37 (1788476137) [ 7768.037350] Lustre: 244533:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7773.157905] Lustre: 244531:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7779.065530] Lustre: server umount lustre-OST0001 complete [ 7786.346442] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 7797.486955] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 7836.044328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7836.340768] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7836.349120] Lustre: Skipped 2 previous similar messages [ 7836.374069] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7836.436422] Lustre: lustre-MDT0000: new disk, initializing [ 7836.488046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7836.500721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7842.032866] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7853.042749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7853.247726] Lustre: 247816:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7853.411660] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7853.415691] Lustre: Skipped 1 previous similar message [ 7853.581302] Lustre: lustre-MDT0001: new disk, initializing [ 7853.775582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7853.820380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7853.838952] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7858.817253] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7869.236232] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7875.559040] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7882.474173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7882.774679] Lustre: lustre-OST0000: new disk, initializing [ 7882.778140] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7882.785757] Lustre: 249788:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7882.862717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7889.728933] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7892.687287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7892.704118] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7892.900562] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7898.770150] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7901.336973] LustreError: 247825:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7909.814484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7910.108463] Lustre: lustre-OST0001: new disk, initializing [ 7910.112873] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7910.124874] Lustre: 250901:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7911.384719] LustreError: 247825:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7911.460812] LustreError: 248950:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7916.411318] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7918.823182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7918.847098] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7918.944515] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7921.519386] LustreError: 248950:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7926.878728] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7928.223684] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7934.454808] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7934.458453] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7934.468822] LustreError: Skipped 1 previous similar message [ 7934.473354] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7934.482056] Lustre: Skipped 2 previous similar messages [ 7936.267902] Lustre: server umount lustre-OST0001 complete [ 7939.571841] LustreError: 250266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7939.596703] LustreError: 250266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7954.401897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7954.403706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7954.411704] LustreError: Skipped 1 previous similar message [ 7954.413085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7954.425873] Lustre: Skipped 1 previous similar message [ 7954.431270] Lustre: Skipped 1 previous similar message [ 7960.039082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7960.041636] Lustre: Skipped 2 previous similar messages [ 7965.160381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7965.167008] Lustre: Skipped 1 previous similar message [ 7968.223250] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7968.373064] Lustre: server umount lustre-OST0000 complete [ 7975.395515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7975.398523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7975.408358] Lustre: Skipped 1 previous similar message [ 7975.421439] Lustre: Skipped 1 previous similar message [ 7978.758752] Lustre: server umount lustre-MDT0000 complete [ 7980.517365] LustreError: 247828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7980.541366] LustreError: 247828:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 7983.221642] LustreError: 247808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788476357 with bad export cookie 18147148930885496119 [ 7983.224632] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7983.228927] LustreError: 247808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7989.679408] Lustre: server umount lustre-MDT0001 complete [ 8000.931163] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 8003.981276] Key type lgssc unregistered [ 8004.306834] LNet: 252935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8004.319608] LNetError: 252935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8004.333363] LNet: Removed LNI 192.168.202.135@tcp [ 8005.244530] Key type .llcrypt unregistered [ 8005.246766] Key type ._llcrypt unregistered [ 8042.251559] Key type ._llcrypt registered [ 8042.254648] Key type .llcrypt registered [ 8042.364119] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 8043.091318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8043.251902] alg: No test for adler32 (adler32-zlib) [ 8044.350948] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 8044.545137] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 8046.167360] Key type lgssc registered [ 8047.217716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8054.132458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8054.160128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8055.583635] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8055.620151] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8055.851582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8060.387349] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8069.369200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8069.503407] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8069.535244] Lustre: 255139:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8069.950714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8075.509124] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8083.485536] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8090.338774] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8096.765672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8096.954209] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8097.102812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8103.668920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8103.724182] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8110.630703] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8165.344834] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8165.350524] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8165.360086] Lustre: Skipped 1 previous similar message [ 8165.370789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8165.378768] Lustre: Skipped 1 previous similar message [ 8170.072551] Lustre: server umount lustre-OST0000 complete [ 8174.165981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8186.458523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8187.018244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8194.119980] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8202.130171] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8205.373087] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8205.384257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8205.388395] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8205.433404] Lustre: Skipped 1 previous similar message [ 8246.240868] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8246.251358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8246.254648] Lustre: Skipped 1 previous similar message [ 8248.596624] Lustre: server umount lustre-OST0000 complete [ 8254.432691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8254.444700] LustreError: Skipped 1 previous similar message [ 8254.448146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8254.463404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8254.468599] Lustre: Skipped 1 previous similar message [ 8256.482749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8256.490959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8261.601639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8261.618665] Lustre: Skipped 2 previous similar messages [ 8267.231156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8267.405858] Lustre: server umount lustre-MDT0000 complete [ 8270.752278] LustreError: 255131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788476644 with bad export cookie 1354704212756533713 [ 8270.755589] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8270.759662] LustreError: 255131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8271.171696] Lustre: server umount lustre-MDT0001 complete [ 8281.372735] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 8284.170781] Key type lgssc unregistered [ 8284.556720] LNet: 260574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8284.576621] LNetError: 260574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8284.591810] LNet: Removed LNI 192.168.202.135@tcp [ 8285.538898] Key type .llcrypt unregistered [ 8285.541454] Key type ._llcrypt unregistered [ 8308.847357] Key type ._llcrypt registered [ 8308.848571] Key type .llcrypt registered [ 8308.954494] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 8310.163295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8310.209771] alg: No test for adler32 (adler32-zlib) [ 8311.366454] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 8311.615353] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 8313.311587] Key type lgssc registered [ 8314.452120] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8321.479992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8321.500750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8323.012096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8327.927891] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8332.266798] LustreError: 261988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8332.285960] LustreError: 261988:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8337.269016] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8337.381624] LustreError: 261987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8337.798805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8342.831413] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8351.133783] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8357.106250] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8364.205574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8364.744794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8367.861927] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8371.199839] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8378.895839] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8403.939605] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8403.951476] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8403.976807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8408.051092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8408.052965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8408.065406] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8410.160607] Lustre: server umount lustre-OST0000 complete [ 8414.695326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8414.706733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8414.722454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8414.726262] Lustre: Skipped 1 previous similar message [ 8418.273271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8418.273589] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8419.955952] Lustre: server umount lustre-MDT0000 complete [ 8423.020273] LustreError: 262721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788476796 with bad export cookie 17747025284526289519 [ 8423.021392] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8423.029110] LustreError: 262721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8423.319671] Lustre: server umount lustre-MDT0001 complete [ 8433.618931] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 8436.458893] Key type lgssc unregistered [ 8436.816971] LNet: 266290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8436.824810] LNetError: 266290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8436.853643] LNet: Removed LNI 192.168.202.135@tcp [ 8437.667218] Key type .llcrypt unregistered [ 8437.668513] Key type ._llcrypt unregistered [ 8453.380160] Key type ._llcrypt registered [ 8453.381987] Key type .llcrypt registered [ 8453.494246] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 8455.047472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8455.129533] alg: No test for adler32 (adler32-zlib) [ 8456.131337] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 8456.387302] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 8458.095568] Key type lgssc registered [ 8459.446183] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8467.753378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8467.781800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8469.306798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8473.991777] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8479.722379] LustreError: 267510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8479.735554] LustreError: 267510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8482.900857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8483.252308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8488.499350] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8495.509657] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8502.110465] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8510.374231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8510.897862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8514.995115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8517.145987] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8523.497945] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8538.596559] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8538.614816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8543.718908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8543.726015] Lustre: Skipped 1 previous similar message [ 8543.936885] Lustre: server umount lustre-OST0000 complete [ 8548.834374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8548.837421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8548.847520] Lustre: Skipped 2 previous similar messages [ 8548.861901] Lustre: Skipped 2 previous similar messages [ 8553.958499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8553.971057] Lustre: Skipped 1 previous similar message [ 8554.548433] Lustre: server umount lustre-MDT0000 complete [ 8559.078413] LustreError: 267511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8559.118907] LustreError: 267511:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8559.225182] LustreError: 267492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788476933 with bad export cookie 6718871279452568091 [ 8559.262141] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8559.913174] Lustre: server umount lustre-MDT0001 complete [ 8573.321318] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 8576.573932] Key type lgssc unregistered [ 8576.863524] LNet: 271188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8576.871650] LNetError: 271188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8576.889783] LNet: Removed LNI 192.168.202.135@tcp [ 8577.662266] Key type .llcrypt unregistered [ 8577.664296] Key type ._llcrypt unregistered [ 8602.564730] Key type ._llcrypt registered [ 8602.570770] Key type .llcrypt registered [ 8602.701126] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 8603.956488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8604.204139] alg: No test for adler32 (adler32-zlib) [ 8605.216479] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 8605.419510] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 8607.055223] Key type lgssc registered [ 8608.466531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8617.737601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8617.782880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8619.375874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8624.914718] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8629.752267] LustreError: 272604:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8629.787064] LustreError: 272604:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8634.008349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8634.474422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8639.918382] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8647.595721] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8653.666660] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8661.157527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8661.754868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8665.858708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8669.257124] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8676.387540] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8706.020303] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8706.040342] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8706.042535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8710.197791] Lustre: server umount lustre-OST0000 complete [ 8716.256769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8716.262062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8716.264771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8716.286447] Lustre: Skipped 2 previous similar messages [ 8716.294303] Lustre: Skipped 1 previous similar message [ 8720.915809] Lustre: server umount lustre-MDT0000 complete [ 8721.377821] LustreError: 273332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8724.561714] LustreError: 272584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788477098 with bad export cookie 9288893421885144011 [ 8724.569781] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8724.579615] LustreError: 272584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8724.941897] Lustre: server umount lustre-MDT0001 complete [ 8734.989808] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 8737.806053] Key type lgssc unregistered [ 8738.149410] LNet: 276909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8738.158461] LNetError: 276909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8739.173855] LNet: Removed LNI 192.168.202.135@tcp [ 8740.276121] Key type .llcrypt unregistered [ 8740.286578] Key type ._llcrypt unregistered [ 8758.098618] Key type ._llcrypt registered [ 8758.100973] Key type .llcrypt registered [ 8758.248678] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 8759.707731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8759.763761] alg: No test for adler32 (adler32-zlib) [ 8760.861168] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 8761.120244] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 8762.791179] Key type lgssc registered [ 8763.799884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8770.882700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8770.916769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8772.411840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8776.712830] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8777.701294] LustreError: 278141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8777.720730] LustreError: 278141:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8782.828210] LustreError: 278143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8784.267643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8784.567441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8788.282383] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8794.903851] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8800.298505] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8806.645711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8807.034457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8812.770626] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8817.339440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8821.346105] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8832.994567] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8832.998698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8833.008265] Lustre: Skipped 1 previous similar message [ 8838.119673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8838.123935] Lustre: Skipped 1 previous similar message [ 8839.057372] Lustre: server umount lustre-OST0000 complete [ 8843.233075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8843.234587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8843.248287] Lustre: Skipped 1 previous similar message [ 8843.259423] Lustre: Skipped 2 previous similar messages [ 8848.353791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8848.371037] Lustre: Skipped 1 previous similar message [ 8848.538193] Lustre: server umount lustre-MDT0000 complete [ 8852.039891] LustreError: 278108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788477225 with bad export cookie 6561592240504168091 [ 8852.047039] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8852.050961] LustreError: 278108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8852.418356] Lustre: server umount lustre-MDT0001 complete [ 8863.004950] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 8865.471737] Key type lgssc unregistered [ 8865.823798] LNet: 281839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8865.839663] LNetError: 281839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8866.853663] LNet: Removed LNI 192.168.202.135@tcp [ 8867.800770] Key type .llcrypt unregistered [ 8867.802153] Key type ._llcrypt unregistered [ 8877.844850] Key type ._llcrypt registered [ 8877.851243] Key type .llcrypt registered [ 8877.981617] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8894.896557] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 8910.187592] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 8911.470801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8911.694842] alg: No test for adler32 (adler32-zlib) [ 8912.804956] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 8913.154190] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 8914.871224] Key type lgssc registered [ 8916.245826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8953.688374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8953.753511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8955.044836] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8955.085783] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8955.179194] Lustre: lustre-MDT0000: new disk, initializing [ 8955.237476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8955.253424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8959.476383] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8970.076199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8970.164174] Lustre: 285393:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8970.222898] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8970.226135] Lustre: Skipped 1 previous similar message [ 8970.280752] Lustre: lustre-MDT0001: new disk, initializing [ 8970.344964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8970.379804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8970.388818] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8974.494207] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8983.336918] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8988.405712] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8994.356172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8994.599618] Lustre: lustre-OST0000: new disk, initializing [ 8994.603678] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8994.609058] Lustre: 287361:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8994.717816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9000.768508] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9002.135124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9002.148764] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9002.294645] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9008.763175] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9013.913308] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9014.138698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9018.649428] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9018.809284] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9022.946195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9022.955501] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9022.977881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9028.075874] Lustre: server umount lustre-OST0000 complete [ 9033.184868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9033.192772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9033.210708] Lustre: Skipped 2 previous similar messages [ 9033.236506] Lustre: Skipped 2 previous similar messages [ 9038.306120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9038.316518] Lustre: Skipped 1 previous similar message [ 9039.258792] Lustre: server umount lustre-MDT0000 complete [ 9043.720850] LustreError: 285384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788477417 with bad export cookie 15900812868456258487 [ 9043.726155] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9044.395592] Lustre: server umount lustre-MDT0001 complete [ 9055.395082] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 19:17:07 (1788477427) [ 9063.500022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9064.319821] LustreError: 289773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9064.347150] LustreError: 289773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9064.478316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9069.568699] LustreError: 289774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9071.669573] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9074.657476] LustreError: 289773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9080.442870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9080.800231] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9080.945726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9084.906777] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9091.975546] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9097.680620] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9104.583724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9104.986652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9111.045618] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9117.712844] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9130.982439] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9130.995280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9130.998112] LustreError: Skipped 1 previous similar message [ 9130.998679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9131.018609] Lustre: Skipped 1 previous similar message [ 9134.813729] Lustre: server umount lustre-OST0000 complete [ 9141.222688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9141.235446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9141.242563] Lustre: Skipped 1 previous similar message [ 9141.247540] Lustre: Skipped 1 previous similar message [ 9144.879534] Lustre: server umount lustre-MDT0000 complete [ 9146.336325] LustreError: 289770:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9146.364832] LustreError: 289770:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9148.453686] LustreError: 289753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788477522 with bad export cookie 15900812868456259383 [ 9148.457619] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9148.467500] LustreError: 289753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 10 previous similar messages [ 9148.820705] Lustre: server umount lustre-MDT0001 complete [ 9159.546937] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 9162.289913] Key type lgssc unregistered [ 9162.708707] LNet: 293308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9162.719524] LNetError: 293308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9162.742472] LNet: Removed LNI 192.168.202.135@tcp [ 9163.845244] Key type .llcrypt unregistered [ 9163.855931] Key type ._llcrypt unregistered [ 9192.357858] Key type ._llcrypt registered [ 9192.362168] Key type .llcrypt registered [ 9192.572785] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 9207.842149] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 9209.392689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9209.516350] alg: No test for adler32 (adler32-zlib) [ 9210.561433] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 9210.792347] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 9212.463474] Key type lgssc registered [ 9213.695724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9262.390853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9262.433881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9264.000604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9264.098714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9264.220271] Lustre: lustre-MDT0000: new disk, initializing [ 9264.288521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9264.320904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9268.681723] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9282.056289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9282.220607] Lustre: 297136:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9282.307246] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9282.314394] Lustre: Skipped 1 previous similar message [ 9282.424465] Lustre: lustre-MDT0001: new disk, initializing [ 9282.516102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9282.579774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9282.602965] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9289.720602] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9301.545180] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9308.426732] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9316.659252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9316.983219] Lustre: lustre-OST2710: new disk, initializing [ 9316.986806] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9316.995124] Lustre: 299104:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9317.097471] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9322.656935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9322.666910] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9322.795972] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9324.648081] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9335.969965] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9342.746958] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9342.862057] Lustre: lustre-OST03e8: new disk, initializing [ 9342.868888] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9342.874310] Lustre: 300168:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9342.959509] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9349.156832] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9353.075282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9353.087605] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9353.305312] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9358.445345] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9364.486269] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9364.704490] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9369.678767] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9369.927475] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9374.679730] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9374.928412] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9379.989966] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9380.242200] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9384.929343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9384.936054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9384.949447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9387.320787] Lustre: server umount lustre-MDT0000 complete [ 9389.028960] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9389.048180] Lustre: Skipped 2 previous similar messages [ 9391.155459] LustreError: 299103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788477765 with bad export cookie 14534440782626409010 [ 9391.156615] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9391.167071] LustreError: 299103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9391.546127] Lustre: server umount lustre-MDT0001 complete [ 9401.569678] Lustre: 302248:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 9405.857406] Lustre: server umount lustre-OST2710 complete [ 9406.945790] Lustre: 302251:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 9410.024389] Lustre: 302248:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 9415.140183] Lustre: 302251:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 9420.272408] Lustre: 302248:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 9420.762633] Lustre: server umount lustre-OST03e8 complete [ 9441.854073] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 9453.008542] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 9497.076339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9497.318245] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9497.337821] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9497.400928] Lustre: lustre-MDT0000: new disk, initializing [ 9497.461117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9497.474524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9501.898249] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9512.466148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9512.526398] Lustre: 305425:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9512.595786] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9512.601697] Lustre: Skipped 1 previous similar message [ 9512.653670] Lustre: lustre-MDT0001: new disk, initializing [ 9512.711799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9512.729247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9512.746750] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9517.376627] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9527.069269] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9532.668271] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9539.616048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9539.961694] Lustre: lustre-OST0000: new disk, initializing [ 9539.964359] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9539.972208] Lustre: 307392:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9540.089913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9546.879301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9546.893536] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9547.085649] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9547.481829] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9557.557216] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9562.646660] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9562.823788] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9567.551387] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9567.753267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9572.843683] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9572.860539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9572.861517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9572.861832] LustreError: Skipped 1 previous similar message [ 9572.887439] Lustre: Skipped 1 previous similar message [ 9575.904823] Lustre: server umount lustre-OST0000 complete [ 9583.073140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9583.085772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9583.088576] Lustre: Skipped 1 previous similar message [ 9585.943876] Lustre: server umount lustre-MDT0000 complete [ 9588.192679] LustreError: 305436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9588.207816] LustreError: 305436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9589.772911] LustreError: 305415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788477963 with bad export cookie 14534440782626410739 [ 9589.776678] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9589.800023] LustreError: 305415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9590.316502] Lustre: server umount lustre-MDT0001 complete [ 9599.761480] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 19:26:12 (1788477972) [ 9601.129795] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9602.806386] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 19:26:15 (1788477975) [ 9607.188740] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 9637.640708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9637.968345] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9637.986738] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9638.221812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9643.139392] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9651.639951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9651.753924] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9651.784761] Lustre: 311249:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9656.947880] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9678.644255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9678.841588] Lustre: Denying initial registration attempt from nid 192.168.202.135@tcp, specified as failover [ 9678.863289] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9678.876846] LustreError: 312704:0:(tgt_mount.c:2734:server_fill_super()) Unable to start targets: -99 [ 9678.886349] LustreError: 312704:0:(tgt_mount.c:2192:server_put_super()) no obd lustre-OST0000 [ 9678.894743] LustreError: 312704:0:(tgt_mount.c:130:server_deregister_mount()) lustre-OST0000 not registered [ 9678.979627] Lustre: server umount lustre-OST0000 complete [ 9678.982998] LustreError: 312704:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9688.033468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9688.054748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9688.082836] Lustre: Skipped 1 previous similar message [ 9688.092508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9693.582598] Lustre: server umount lustre-MDT0000 complete [ 9695.723365] LustreError: 311257:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9695.749905] LustreError: 311257:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9698.421950] LustreError: 311242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788478072 with bad export cookie 14534440782626411635 [ 9698.425035] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9698.444689] LustreError: 311242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9698.654529] Lustre: server umount lustre-MDT0001 complete [ 9712.891387] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 9715.959576] Key type lgssc unregistered [ 9716.311538] LNet: 313822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9716.322867] LNetError: 313822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9717.352935] LNet: Removed LNI 192.168.202.135@tcp [ 9718.274310] Key type .llcrypt unregistered [ 9718.277150] Key type ._llcrypt unregistered [ 9729.202822] Key type ._llcrypt registered [ 9729.205139] Key type .llcrypt registered [ 9729.286141] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 9730.322206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9730.381015] alg: No test for adler32 (adler32-zlib) [ 9731.467576] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 9731.775152] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 9733.463758] Key type lgssc registered [ 9734.767350] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9780.699113] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 9789.853696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9789.866950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9791.123225] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9791.157611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9791.454419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9796.059282] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9804.863956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9805.023299] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9805.096305] Lustre: 316583:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9805.542186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9810.433960] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9818.582704] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9825.240651] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9846.191417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9846.460243] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9846.707295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9853.886247] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9862.053388] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9868.770370] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9868.777935] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9868.801729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9869.288201] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9869.293720] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9869.311462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9869.316408] Lustre: Skipped 1 previous similar message [ 9870.599519] Lustre: server umount lustre-OST0000 complete [ 9877.473392] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9877.487577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9877.508457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9879.521089] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9879.523393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9879.546763] Lustre: Skipped 1 previous similar message [ 9881.054951] Lustre: server umount lustre-MDT0000 complete [ 9884.580884] LustreError: 316577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788478258 with bad export cookie 13979360474367015566 [ 9884.588691] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9884.592310] LustreError: 316577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9885.048388] Lustre: server umount lustre-MDT0001 complete [ 9896.545528] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 9899.310723] Key type lgssc unregistered [ 9899.705488] LNet: 320161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9899.713463] LNetError: 320161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9899.733073] LNet: Removed LNI 192.168.202.135@tcp [ 9900.572281] Key type .llcrypt unregistered [ 9900.574490] Key type ._llcrypt unregistered [ 9929.501535] Key type ._llcrypt registered [ 9929.503163] Key type .llcrypt registered [ 9929.695876] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 9931.050931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9931.085100] alg: No test for adler32 (adler32-zlib) [ 9932.570747] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 9932.987799] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 9934.631360] Key type lgssc registered [ 9936.136400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9944.959796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9944.985690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9946.529548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9951.931376] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9956.835042] LustreError: 321579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9956.877726] LustreError: 321579:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9960.362962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9960.697734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9966.028360] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9975.149518] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9981.554095] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9990.960667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9991.356142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9997.840867] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10005.080541] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10012.800234] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.35@tcp (stopping) [10016.239612] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10016.255470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10017.863901] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.35@tcp (stopping) [10017.876563] Lustre: Skipped 2 previous similar messages [10019.150231] Lustre: server umount lustre-MDT0000 complete [10021.354566] LustreError: 321579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10021.375993] LustreError: 321579:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10022.988643] LustreError: 321573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10023.189689] LustreError: 321603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788478397 with bad export cookie 15516426818714187453 [10023.190741] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10023.197676] LustreError: 321603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10026.464612] LustreError: 321575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10026.464862] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10026.492657] Lustre: Skipped 2 previous similar messages [10026.500288] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10029.730050] Lustre: server umount lustre-MDT0001 complete [10033.635219] Lustre: 324368:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [10036.270437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10038.752550] Lustre: 324368:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [10043.881479] Lustre: 324369:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [10044.562662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10045.084121] LustreError: 323412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10045.109488] LustreError: 323412:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10045.174141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10045.245920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10050.080558] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10053.604409] LustreError: 325179:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10053.630260] LustreError: 325179:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10058.089542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10058.507070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10059.580314] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10063.663725] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10071.248980] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10072.810797] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10078.957486] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10081.169386] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10086.369876] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10086.381304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10086.404601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10086.412630] Lustre: Skipped 2 previous similar messages [10089.442533] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10089.448207] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10095.585608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10095.598231] Lustre: Skipped 4 previous similar messages [10100.192237] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10100.516280] Lustre: server umount lustre-OST0000 complete [10104.802617] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10104.807155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10110.834383] Lustre: server umount lustre-MDT0000 complete [10110.962144] LustreError: 325183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10110.985663] LustreError: 325183:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10114.897809] LustreError: 321603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788478488 with bad export cookie 15516426818714195293 [10114.906841] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10114.908551] LustreError: 321603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10115.539443] Lustre: server umount lustre-MDT0001 complete [10129.822111] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [10133.351520] Key type lgssc unregistered [10133.802717] LNet: 327855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10133.810408] LNetError: 327855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10133.827457] LNet: Removed LNI 192.168.202.135@tcp [10134.640316] Key type .llcrypt unregistered [10134.646956] Key type ._llcrypt unregistered [10162.649267] Key type ._llcrypt registered [10162.654902] Key type .llcrypt registered [10162.801695] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [10164.200743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10164.357485] alg: No test for adler32 (adler32-zlib) [10165.489533] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [10165.847249] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [10167.567128] Key type lgssc registered [10169.157474] Lustre: Echo OBD driver; http://www.lustre.org/ [10176.754342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10176.768238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10178.543585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10183.927642] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10188.771456] LustreError: 329270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10188.778349] LustreError: 329270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10191.939357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10192.213427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10197.883933] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10206.137775] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10213.415579] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10217.507169] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10217.511285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10217.523424] Lustre: Skipped 1 previous similar message [10217.543152] Lustre: Skipped 1 previous similar message [10222.562288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10222.571404] Lustre: Skipped 1 previous similar message [10222.781437] Lustre: server umount lustre-MDT0000 complete [10226.634943] LustreError: 329251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788478600 with bad export cookie 18033945157096035725 [10226.645230] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10226.647634] LustreError: 329251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10226.973758] Lustre: server umount lustre-MDT0001 complete [10244.255897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10244.613288] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10244.646183] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10244.896984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10249.917655] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10260.745238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10260.894571] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10260.987308] Lustre: 331859:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10261.391543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10267.244841] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10275.142267] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10281.616434] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10292.601578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10292.851299] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10292.859918] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10292.874585] LustreError: 333739:0:(tgt_mount.c:2734:server_fill_super()) Unable to start targets: -2 [10292.883254] LustreError: 333739:0:(tgt_mount.c:2192:server_put_super()) no obd lustre-OST0000 [10292.888178] LustreError: 333739:0:(tgt_mount.c:130:server_deregister_mount()) lustre-OST0000 not registered [10293.016485] Lustre: server umount lustre-OST0000 complete [10293.024875] LustreError: 333739:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10304.654323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10304.972874] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10305.153020] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10311.469807] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10314.479819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10319.293763] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10326.805042] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10326.889823] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10327.003273] Lustre: lustre-OST0001: new disk, initializing [10327.008643] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10327.022828] Lustre: 335282:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10327.124705] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10335.740967] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10336.017422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10336.036944] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10336.247159] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10344.734910] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10351.592472] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10351.592519] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10351.613426] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10351.618698] LustreError: Skipped 1 previous similar message [10351.643733] Lustre: Skipped 1 previous similar message [10354.497691] Lustre: server umount lustre-OST0001 complete [10356.715593] LustreError: 334208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10356.755759] LustreError: 334208:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10360.801633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10360.814041] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10360.820181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10360.828957] Lustre: Skipped 2 previous similar messages [10360.850303] Lustre: Skipped 1 previous similar message [10361.830855] LustreError: 334206:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10361.855661] LustreError: 334206:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10365.283854] Lustre: server umount lustre-OST0000 complete [10368.992902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10369.003701] LustreError: Skipped 1 previous similar message [10369.005397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10369.023848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10369.028138] Lustre: Skipped 2 previous similar messages [10377.184982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10377.194589] Lustre: Skipped 3 previous similar messages [10383.327197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10383.561840] Lustre: server umount lustre-MDT0000 complete [10386.412595] LustreError: 331868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10386.454168] LustreError: 331868:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10387.723225] LustreError: 331850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788478761 with bad export cookie 18033945157096036292 [10387.727765] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10387.734900] LustreError: 331850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10388.069523] Lustre: server umount lustre-MDT0001 complete [10400.317562] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [10403.825979] Key type lgssc unregistered [10404.209351] LNet: 337242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10404.217503] LNetError: 337242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10404.248519] LNet: Removed LNI 192.168.202.135@tcp [10405.223655] Key type .llcrypt unregistered [10405.226402] Key type ._llcrypt unregistered [10480.471812] Key type ._llcrypt registered [10480.487463] Key type .llcrypt registered [10480.639259] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [10495.489860] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [10496.594352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10496.693050] alg: No test for adler32 (adler32-zlib) [10497.818780] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [10498.290424] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [10499.984668] Key type lgssc registered [10501.249912] Lustre: Echo OBD driver; http://www.lustre.org/ [10533.083621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10533.116688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10534.338622] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10534.364551] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10534.459281] Lustre: lustre-MDT0000: new disk, initializing [10534.537537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10534.554958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10539.525789] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10551.390737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10551.551709] Lustre: 342177:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10551.650771] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10551.664147] Lustre: Skipped 1 previous similar message [10551.774935] Lustre: lustre-MDT0001: new disk, initializing [10551.886177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10551.931703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10551.949041] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10556.611905] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10566.317715] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10573.527794] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10581.485323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10581.746199] Lustre: lustre-OST0000: new disk, initializing [10581.748498] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10581.753366] Lustre: 344146:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10581.823404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10588.370469] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10588.859389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10588.882500] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10589.003752] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10599.412525] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10605.579811] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10605.852795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10612.789437] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10613.180630] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10619.875926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10619.878729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10619.920496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10619.935774] Lustre: Skipped 1 previous similar message [10619.960633] Lustre: Skipped 1 previous similar message [10622.533084] Lustre: server umount lustre-OST0000 complete [10628.578906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10628.589259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10628.601393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10630.118716] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10630.120052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10630.148894] Lustre: Skipped 1 previous similar message [10634.853256] Lustre: server umount lustre-MDT0000 complete [10638.731288] LustreError: 342168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788479012 with bad export cookie 12412983415225022372 [10638.740609] LustreError: 342168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10638.742145] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10639.429077] Lustre: server umount lustre-MDT0001 complete [10648.398264] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 19:43:41 (1788479021) [10655.996474] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 19:43:48 (1788479028) [10665.420688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10665.883493] LustreError: 346901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10665.915029] LustreError: 346901:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10665.996259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10671.076325] LustreError: 346902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10671.181276] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10675.170095] LustreError: 346901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10680.288295] LustreError: 346902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10680.750131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10681.149209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10686.534776] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10694.522187] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10700.681598] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10707.108894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10707.518297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10713.739955] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10721.923669] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10734.564839] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10736.553305] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10738.361422] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10739.929969] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10741.669563] Lustre: Failing over lustre-MDT0000 [10741.941530] Lustre: server umount lustre-MDT0000 complete [10742.753464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10742.761322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10742.771464] LustreError: 349518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10751.462262] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10751.467244] LustreError: 346898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10751.500261] Lustre: 349813:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [10751.542678] LustreError: 346898:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10756.585553] Lustre: 349816:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [10760.900770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10761.289173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10761.358259] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10765.912527] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10766.328901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10766.366546] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10766.430658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10766.781646] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10776.921642] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10779.052565] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10782.848550] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10785.164554] Lustre: Failing over lustre-MDT0000 [10785.579750] Lustre: server umount lustre-MDT0000 complete [10786.419191] LustreError: 346896:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10786.445490] LustreError: 346896:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10786.787889] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10786.793393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10786.812942] Lustre: Skipped 3 previous similar messages [10797.031260] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10797.080434] Lustre: 351019:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [10802.159735] Lustre: 351019:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [10807.073759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10807.271951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [10807.390396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10807.439369] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10811.979840] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10812.416591] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10812.427560] Lustre: Skipped 2 previous similar messages [10812.481775] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10812.541038] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10813.273300] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10824.981286] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10826.829500] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10832.872525] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10832.875020] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10832.881753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10832.881763] Lustre: Skipped 2 previous similar messages [10832.883439] Lustre: Skipped 1 previous similar message [10838.987657] Lustre: server umount lustre-MDT0000 complete [10843.110392] LustreError: 346901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10843.129968] LustreError: 346901:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [10843.138938] LustreError: 346883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788479217 with bad export cookie 12412983415225025249 [10843.144501] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10843.202791] LustreError: 346883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10843.502691] Lustre: server umount lustre-MDT0001 complete [10853.541950] Lustre: 352337:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [10857.846561] Lustre: server umount lustre-OST0000 complete [10873.298974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10873.797114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10878.490503] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10887.434677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10893.803756] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10901.684442] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10907.584260] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10914.505571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10918.912388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10921.390098] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10928.927167] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10939.370151] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10939.374922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10939.404436] Lustre: Skipped 1 previous similar message [10939.416046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10939.418739] Lustre: Skipped 5 previous similar messages [10945.214021] Lustre: server umount lustre-OST0000 complete [10956.040374] Lustre: server umount lustre-MDT0000 complete [10957.798356] LustreError: 353282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10957.827142] LustreError: 353282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10960.176463] LustreError: 353267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788479334 with bad export cookie 12412983415225026033 [10960.188623] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10960.198516] LustreError: 353267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10960.756956] Lustre: server umount lustre-MDT0001 complete [10971.923549] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [10975.402598] Key type lgssc unregistered [10975.829449] LNet: 356817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10975.839650] LNetError: 356817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10975.874492] LNet: Removed LNI 192.168.202.135@tcp [10977.168276] Key type .llcrypt unregistered [10977.170360] Key type ._llcrypt unregistered [11004.704837] Key type ._llcrypt registered [11004.707473] Key type .llcrypt registered [11004.820886] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [11005.861615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11005.914899] alg: No test for adler32 (adler32-zlib) [11007.339327] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [11007.531582] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [11009.207280] Key type lgssc registered [11010.729966] Lustre: Echo OBD driver; http://www.lustre.org/ [11018.710494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11018.736441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11020.274801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11025.315644] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11030.499499] LustreError: 358332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11030.532325] LustreError: 358332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11034.573955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11035.101402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11039.529695] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11046.347338] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11051.898730] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11058.829452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11059.160919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11065.363292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11065.586978] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11074.119278] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11087.226749] Lustre: server umount lustre-MDT0000 complete [11090.916460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11090.929183] LustreError: 358328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11090.943328] Lustre: Skipped 2 previous similar messages [11090.961224] LustreError: 358328:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11092.395525] LustreError: 358313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788479466 with bad export cookie 17780527337971163925 [11092.398912] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11092.403429] LustreError: 358313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11092.740838] Lustre: server umount lustre-MDT0001 complete [11102.304818] Lustre: 361263:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [11107.816450] Lustre: 361265:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [11109.378108] Lustre: server umount lustre-OST0000 complete [11130.692475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11131.443715] LustreError: 362065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11131.463635] LustreError: 362065:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11131.528428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11136.662739] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11136.994930] LustreError: 362066:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11142.114342] LustreError: 362065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11145.645899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11146.156501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11151.212308] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11159.910891] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11167.316898] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11174.114389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11174.506456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11177.610807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11182.935774] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11191.158828] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11206.632261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11206.633435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11206.648664] Lustre: Skipped 1 previous similar message [11206.664770] Lustre: Skipped 2 previous similar messages [11211.088620] Lustre: server umount lustre-MDT0000 complete [11211.750444] LustreError: 362065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11211.774523] LustreError: 362065:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11215.908225] LustreError: 362048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788479589 with bad export cookie 17780527337971165087 [11215.910124] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11215.921315] LustreError: 362048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11216.561909] Lustre: server umount lustre-MDT0001 complete [11225.889000] Lustre: 365063:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [11231.243545] Lustre: server umount lustre-OST0000 complete [11251.657316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11252.321566] LustreError: 365982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11252.438650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11257.734150] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11267.772250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11274.265865] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11283.364849] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11289.453549] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11298.832095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11299.282350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11299.287208] Lustre: Skipped 1 previous similar message [11306.517082] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11314.681824] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11322.825838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11333.090238] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11333.093530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11333.113120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11333.134549] Lustre: Skipped 2 previous similar messages [11338.208977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11338.223591] Lustre: Skipped 2 previous similar messages [11343.331770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11343.349323] Lustre: Skipped 1 previous similar message [11346.399187] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11346.514737] Lustre: server umount lustre-OST0000 complete [11350.496464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11350.504303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11350.520777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11356.541150] Lustre: server umount lustre-MDT0000 complete [11358.690669] LustreError: 367270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11358.714469] LustreError: 367270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11361.065770] LustreError: 365963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788479734 with bad export cookie 17780527337971166151 [11361.073474] LustreError: 365963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11361.074507] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11361.425953] Lustre: server umount lustre-MDT0001 complete [11372.442137] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [11375.169556] Key type lgssc unregistered [11375.532309] LNet: 369538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11375.543113] LNetError: 369538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11375.574612] LNet: Removed LNI 192.168.202.135@tcp [11376.593187] Key type .llcrypt unregistered [11376.597939] Key type ._llcrypt unregistered [11406.573377] Key type ._llcrypt registered [11406.577069] Key type .llcrypt registered [11406.704887] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [11408.032792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11408.083571] alg: No test for adler32 (adler32-zlib) [11409.319518] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [11409.816320] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [11411.711162] Key type lgssc registered [11413.592654] Lustre: Echo OBD driver; http://www.lustre.org/ [11421.684267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11421.706920] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11421.714229] LustreError: 370979:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11421.732172] LustreError: 370979:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11421.738520] LustreError: 370979:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11421.743398] LustreError: 370979:0:(tgt_mount.c:2664:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11421.749247] LustreError: 370979:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11429.896544] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11429.901573] LustreError: 371378:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11429.913996] LustreError: 371378:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11429.918308] LustreError: 371378:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11429.921464] LustreError: 371378:0:(tgt_mount.c:2664:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11429.927917] LustreError: 371378:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11449.910827] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [11452.708435] Key type lgssc unregistered [11453.004993] LNet: 372380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11453.011984] LNetError: 372380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11453.027944] LNet: Removed LNI 192.168.202.135@tcp [11454.163524] Key type .llcrypt unregistered [11454.165612] Key type ._llcrypt unregistered [11472.530401] Key type ._llcrypt registered [11472.533600] Key type .llcrypt registered [11472.661298] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [11487.884823] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [11488.841096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11489.013643] alg: No test for adler32 (adler32-zlib) [11490.114098] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [11490.392452] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [11492.079919] Key type lgssc registered [11493.304953] Lustre: Echo OBD driver; http://www.lustre.org/ [11527.927502] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11527.946360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11529.246489] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11529.289833] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11529.441348] Lustre: lustre-MDT0000: new disk, initializing [11529.515669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11529.529412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11534.015460] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11543.926854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11544.042571] Lustre: 375535:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11544.164950] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11544.178443] Lustre: Skipped 1 previous similar message [11544.262250] Lustre: lustre-MDT0001: new disk, initializing [11544.383139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11544.442112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11544.463646] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11548.786920] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11558.380671] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11563.491302] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11570.123315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11570.457193] Lustre: lustre-OST0000: new disk, initializing [11570.460725] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11570.466290] Lustre: 377499:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11570.556540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11576.878417] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11580.562377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11580.575063] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11580.752574] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11587.645547] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11592.161329] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11592.369856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11597.191860] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11597.408793] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11601.382067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11601.383533] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11601.385116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11601.397025] Lustre: Skipped 1 previous similar message [11601.414803] LustreError: Skipped 1 previous similar message [11605.722415] Lustre: server umount lustre-OST0000 complete [11611.118061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11611.132557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11611.154205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11611.164285] Lustre: Skipped 1 previous similar message [11615.937448] Lustre: server umount lustre-MDT0000 complete [11619.405566] LustreError: 375526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788479993 with bad export cookie 3608973968856739203 [11619.414607] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11619.422871] LustreError: 375526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11619.955504] Lustre: server umount lustre-MDT0001 complete [11629.565514] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11631.369327] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 20:00:04 (1788480004) [11638.625858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11639.168374] LustreError: 379962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11639.204246] LustreError: 379962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11639.324430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11644.390071] LustreError: 379963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11646.706947] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11649.507712] LustreError: 379962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11654.627954] LustreError: 379963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11657.969593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11658.461294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11664.648214] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11672.976550] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11678.171290] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11686.244428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11686.542504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11692.872227] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11700.785175] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11707.387321] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11707.609738] Lustre: lustre-OST0001: new disk, initializing [11707.613077] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11707.618448] Lustre: 382811:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11707.698434] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11714.078828] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11715.278601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11715.293750] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11715.392213] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11723.377788] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11730.070498] Lustre: lustre-OST0001: Not available for connect from 192.168.202.35@tcp (stopping) [11730.086391] Lustre: Skipped 1 previous similar message [11730.917244] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11730.921722] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11730.929390] Lustre: Skipped 1 previous similar message [11732.834596] Lustre: server umount lustre-OST0001 complete [11735.114164] LustreError: 383243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11746.284470] LustreError: 381786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11746.296792] LustreError: 381786:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11750.374320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11750.387915] LustreError: Skipped 1 previous similar message [11750.395942] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11750.425286] Lustre: Skipped 1 previous similar message [11750.435933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11750.458717] Lustre: Skipped 2 previous similar messages [11755.663896] Lustre: server umount lustre-OST0000 complete [11761.120258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11761.129409] LustreError: Skipped 1 previous similar message [11761.133538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11761.145848] Lustre: Skipped 1 previous similar message [11761.149858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11761.154894] Lustre: Skipped 3 previous similar messages [11766.127313] Lustre: server umount lustre-MDT0000 complete [11766.758638] LustreError: 379957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11766.774538] LustreError: 379957:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11770.193570] LustreError: 379943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788480144 with bad export cookie 3608973968856740099 [11770.215440] LustreError: 379943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11770.222791] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11776.730843] Lustre: server umount lustre-MDT0001 complete [11787.982166] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [11791.082735] Key type lgssc unregistered [11791.429266] LNet: 384800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11791.434299] LNetError: 384800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11791.457395] LNet: Removed LNI 192.168.202.135@tcp [11792.289225] Key type .llcrypt unregistered [11792.290926] Key type ._llcrypt unregistered [11844.400456] Key type ._llcrypt registered [11844.402728] Key type .llcrypt registered [11844.608323] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [11846.928861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11846.985453] alg: No test for adler32 (adler32-zlib) [11848.262560] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [11848.690993] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [11850.423173] Key type lgssc registered [11852.633087] Lustre: Echo OBD driver; http://www.lustre.org/ [11861.006414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11861.033354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11862.354396] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11862.380540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11862.499917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11867.076247] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11876.244924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11876.395675] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11876.437590] Lustre: 387598:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11876.919183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11882.129804] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11892.443894] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11900.275968] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11902.437272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11902.461557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11902.473612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11904.482619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11904.483668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11908.762974] Lustre: server umount lustre-MDT0000 complete [11912.698913] LustreError: 387589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788480286 with bad export cookie 8857709383312036621 [11912.704371] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11912.713770] LustreError: 387589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11913.188259] Lustre: server umount lustre-MDT0001 complete [11923.376827] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 20:04:55 (1788480295) [11930.795861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11931.417861] LustreError: 390112:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11931.441658] LustreError: 390112:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11931.501084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11936.742418] LustreError: 390113:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11936.921947] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11941.859746] LustreError: 390112:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11946.977679] LustreError: 390113:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11947.629640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11948.128360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11952.242245] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11959.787601] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11966.254172] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11974.656136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11974.797970] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11975.178211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11981.608193] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11989.326638] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11997.939949] Lustre: Permanently deactivating lustre-OST0000 [11997.950775] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12006.525711] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12008.361982] LustreError: 393097:0:(mgs_llog.c:2049:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12008.371906] LustreError: 393097:0:(mgs_llog.c:2104:mgs_replace_nids()) Only MGS is allowed to be started [12008.383588] LustreError: 393097:0:(mgs_handler.c:1257:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12012.281436] LustreError: 393146:0:(mgs_llog.c:2104:mgs_replace_nids()) Only MGS is allowed to be started [12012.285827] LustreError: 393146:0:(mgs_handler.c:1257:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12028.383116] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12028.592991] Lustre: server umount lustre-OST0000 complete [12031.251149] LustreError: 393400:0:(mgs_llog.c:2104:mgs_replace_nids()) Only MGS is allowed to be started [12031.257850] LustreError: 393400:0:(mgs_handler.c:1257:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12034.019563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12034.022406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12034.031314] Lustre: Skipped 1 previous similar message [12034.037971] Lustre: Skipped 2 previous similar messages [12039.143380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12039.151055] Lustre: Skipped 1 previous similar message [12043.303907] Lustre: server umount lustre-MDT0000 complete [12044.260370] LustreError: 392122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12044.283886] LustreError: 392122:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12046.997174] LustreError: 390092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788480420 with bad export cookie 8857709383312037202 [12047.009341] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12047.012948] LustreError: 390092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12056.798459] Lustre: server umount lustre-MDT0001 complete [12066.868799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12073.275946] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12077.030578] LustreError: 394560:0:(mgs_handler.c:1257:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12083.728471] Lustre: 394899:0:(mgs_llog.c:1814:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.135@tcp [12085.976947] Lustre: server umount MGS complete [12098.807072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12099.247286] LustreError: 395623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12099.275433] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12099.280871] Lustre: Skipped 1 previous similar message [12099.334919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12104.247103] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12113.136338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12113.535794] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12117.825354] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12125.285810] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12131.799171] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12133.945642] Lustre: Permanently reactivating lustre-OST0000 [12134.012130] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12134.023935] Lustre: Skipped 2 previous similar messages [12159.967334] LustreError: 397356:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12168.241385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12168.644088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12168.662291] Lustre: Skipped 1 previous similar message [12175.007717] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12183.417431] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12195.308284] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12195.328685] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12195.348867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12195.350814] Lustre: Skipped 1 previous similar message [12195.372874] Lustre: Skipped 1 previous similar message [12196.803280] Lustre: server umount lustre-OST0000 complete [12205.537526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12205.554121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12205.576088] Lustre: Skipped 1 previous similar message [12216.287328] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12216.632990] Lustre: server umount lustre-MDT0000 complete [12220.902957] LustreError: 395624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12220.921068] LustreError: 395624:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12221.718693] LustreError: 395605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788480595 with bad export cookie 8857709383312038308 [12221.721302] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12221.723425] LustreError: 395605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12222.051102] Lustre: server umount lustre-MDT0001 complete [12235.305715] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [12238.162691] Key type lgssc unregistered [12238.529246] LNet: 399943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12238.541603] LNetError: 399943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12238.580978] LNet: Removed LNI 192.168.202.135@tcp [12239.597715] Key type .llcrypt unregistered [12239.602110] Key type ._llcrypt unregistered [12259.532906] Key type ._llcrypt registered [12259.535316] Key type .llcrypt registered [12259.622655] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [12276.341723] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [12277.629654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12277.652223] alg: No test for adler32 (adler32-zlib) [12278.752816] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [12279.116108] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [12280.847250] Key type lgssc registered [12282.249199] Lustre: Echo OBD driver; http://www.lustre.org/ [12331.429066] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12331.503188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12333.189594] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12333.277865] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12333.349290] Lustre: lustre-MDT0000: new disk, initializing [12333.473746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12333.492694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12339.366959] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12350.279794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12350.442496] Lustre: 403097:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12350.558854] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12350.561538] Lustre: Skipped 1 previous similar message [12350.650580] Lustre: lustre-MDT0001: new disk, initializing [12350.800865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12350.836919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12350.845889] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12355.393423] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12364.689493] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12369.701951] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12376.717345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12377.062911] Lustre: lustre-OST0000: new disk, initializing [12377.066458] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12377.073642] Lustre: 405069:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12377.176892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12383.500809] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12386.936416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12386.948221] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12387.073051] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12393.527305] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12398.349215] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12398.612882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12403.652712] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12403.919364] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12407.777097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12407.789618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12407.799330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12412.238627] Lustre: server umount lustre-OST0000 complete [12417.510541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12417.515832] LustreError: Skipped 1 previous similar message [12417.519304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12417.536230] Lustre: Skipped 1 previous similar message [12417.545945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12417.553229] Lustre: Skipped 1 previous similar message [12423.136483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12423.142428] Lustre: Skipped 2 previous similar messages [12427.234266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12427.241873] Lustre: Skipped 1 previous similar message [12430.303135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12430.513552] Lustre: server umount lustre-MDT0000 complete [12433.889903] LustreError: 405067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788480807 with bad export cookie 2476246873823951365 [12433.893103] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12433.908642] LustreError: 405067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12434.180425] Lustre: server umount lustre-MDT0001 complete [12442.507380] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12444.199696] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 20:13:36 (1788480816) [12450.351457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12450.698871] LustreError: 407533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12450.715836] LustreError: 407533:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12450.754877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12455.555728] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12455.907858] LustreError: 407534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12461.027102] LustreError: 407533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12463.458848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12463.795662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12468.274838] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12476.399656] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12481.829813] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12489.969449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12490.293714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12497.913982] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12509.193398] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12511.725373] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12514.374319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12520.941768] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12520.948887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12520.978508] Lustre: Skipped 1 previous similar message [12520.996355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12524.017253] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12524.026547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12526.512798] Lustre: server umount lustre-OST0000 complete [12532.192889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12532.225380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12532.231541] Lustre: Skipped 3 previous similar messages [12537.314210] LustreError: 407533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12537.538280] Lustre: server umount lustre-MDT0000 complete [12541.408417] LustreError: 408261:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12541.442000] LustreError: 408261:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12542.711328] LustreError: 407513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788480916 with bad export cookie 2476246873823952261 [12542.721134] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12542.723599] LustreError: 407513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12542.984430] Lustre: server umount lustre-MDT0001 complete [12559.859986] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [12563.424362] Key type lgssc unregistered [12563.873857] LNet: 411258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12563.877314] LNetError: 411258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12563.906176] LNet: Removed LNI 192.168.202.135@tcp [12564.842643] Key type .llcrypt unregistered [12564.844677] Key type ._llcrypt unregistered [12594.294872] Key type ._llcrypt registered [12594.299721] Key type .llcrypt registered [12594.482208] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [12595.010844] Key type .llcrypt unregistered [12595.023461] Key type ._llcrypt unregistered [12618.572592] Key type ._llcrypt registered [12618.577147] Key type .llcrypt registered [12618.696630] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [12619.839140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12620.166268] alg: No test for adler32 (adler32-zlib) [12621.344589] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [12621.678600] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [12623.378320] Key type lgssc registered [12624.333113] Lustre: Echo OBD driver; http://www.lustre.org/ [12634.605927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12634.649370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12636.493854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12641.193408] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12646.894227] LustreError: 413465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12646.903796] LustreError: 413465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12649.203974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12649.458446] LustreError: 413466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12649.480915] LustreError: 413466:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12649.498060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12654.571643] LustreError: 413465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12654.598851] LustreError: 413465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12657.089760] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12659.685163] LustreError: 413459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12659.716843] LustreError: 413459:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12661.760409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12666.061847] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12666.852686] LustreError: 414660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12666.891285] LustreError: 414660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12673.049656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12673.548857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12678.148410] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12692.450858] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12692.455073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12692.481394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12693.990322] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12693.996615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12694.006261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12694.024491] Lustre: Skipped 1 previous similar message [12695.857351] Lustre: server umount lustre-OST0000 complete [12704.224469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12704.228306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12704.246440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12706.328608] Lustre: server umount lustre-MDT0000 complete [12709.346258] LustreError: 413462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12709.346258] LustreError: 413466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12709.346274] LustreError: 413462:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12709.954237] LustreError: 413447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788481083 with bad export cookie 13202659042932040103 [12709.962546] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12709.968802] LustreError: 413447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12710.297249] Lustre: server umount lustre-MDT0001 complete [12721.155875] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [12723.625975] Key type lgssc unregistered [12723.853144] LNet: 416615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12723.860121] LNetError: 416615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12723.872369] LNet: Removed LNI 192.168.202.135@tcp [12724.551442] Key type .llcrypt unregistered [12724.553788] Key type ._llcrypt unregistered [12748.760944] Key type ._llcrypt registered [12748.763988] Key type .llcrypt registered [12748.841816] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [12750.236721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12750.333799] alg: No test for adler32 (adler32-zlib) [12751.477696] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [12751.818130] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [12753.641695] Key type lgssc registered [12755.606197] Lustre: Echo OBD driver; http://www.lustre.org/ [12765.840224] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12765.862641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12766.986688] Lustre: 418022:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [12782.431391] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12787.551445] LustreError: 418010:0:(mgc_request.c:317:do_config_log_add()) MGC192.168.202.135@tcp: failed processing log, type 4: rc = -110 [12792.288383] Lustre: 418025:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [12803.002148] Lustre: 418022:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [12808.160111] Lustre: 418025:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [12812.258905] Lustre: 418022:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [12813.280857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12818.586953] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12822.497767] Lustre: 418022:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [12822.512872] Lustre: 418022:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 1 previous similar message [12827.581489] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12834.529402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12835.004730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12835.062043] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12839.097477] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12840.419057] LustreError: 418040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12840.451392] LustreError: 418040:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12845.557244] LustreError: 418041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12847.605359] LustreError: 419053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12847.631558] LustreError: 419053:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12848.492808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12848.987878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12853.430070] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12861.084578] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12868.170792] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12874.724254] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12874.729264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12874.734761] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12874.738595] Lustre: Skipped 1 previous similar message [12874.765518] Lustre: Skipped 1 previous similar message [12875.369171] Lustre: server umount lustre-OST0000 complete [12879.845323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12879.849983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12879.850975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12879.868160] Lustre: Skipped 1 previous similar message [12884.977147] LustreError: 419058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12885.003615] LustreError: 419058:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12885.182275] Lustre: server umount lustre-MDT0000 complete [12889.403501] LustreError: 418017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788481263 with bad export cookie 16729135638518501140 [12889.421167] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12889.424160] LustreError: 418017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12889.764564] Lustre: server umount lustre-MDT0001 complete [12900.418843] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [12903.191667] Key type lgssc unregistered [12903.528724] LNet: 421584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12903.540647] LNetError: 421584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12903.554732] LNet: Removed LNI 192.168.202.135@tcp [12904.291600] Key type .llcrypt unregistered [12904.294631] Key type ._llcrypt unregistered [12929.868917] Key type ._llcrypt registered [12929.871683] Key type .llcrypt registered [12930.007838] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [12930.919918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12930.957550] alg: No test for adler32 (adler32-zlib) [12932.164372] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [12932.480681] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [12934.122431] Key type lgssc registered [12935.407142] Lustre: Echo OBD driver; http://www.lustre.org/ [12942.085206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12942.105721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12943.447336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12947.358798] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12953.571123] LustreError: 423012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12953.592643] LustreError: 423012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12955.891663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12956.443850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12961.224082] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12968.770378] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12974.767760] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12981.602731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12981.868925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12988.440961] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12989.064301] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12996.035682] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13001.830814] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13001.860031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13001.864227] Lustre: Skipped 1 previous similar message [13001.900565] Lustre: Skipped 2 previous similar messages [13003.245109] Lustre: server umount lustre-MDT0000 complete [13003.896429] LustreError: 423007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13005.867086] LustreError: 422993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.35@tcp arrived at 1788481379 with bad export cookie 9009557122047192235 [13007.337644] LustreError: 423012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13007.349211] LustreError: 423012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13007.580286] LustreError: 422992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788481381 with bad export cookie 9009557122047191318 [13007.598675] LustreError: 422992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13007.604862] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13009.893834] Lustre: server umount lustre-OST0000 complete [13012.449530] LustreError: 423013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13012.469901] LustreError: 423013:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13017.575991] Lustre: 425914:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [13017.584803] LustreError: 423740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13017.637957] LustreError: 423740:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13018.338410] LustreError: 423742:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000cf9e21a5 x1875358573952384/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13022.688606] Lustre: 425916:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [13027.809219] Lustre: 425914:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [13028.186290] Lustre: server umount lustre-MDT0001 complete [13039.368207] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [13041.804503] Key type lgssc unregistered [13042.220699] LNet: 426701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13042.232606] LNetError: 426701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13042.260212] LNet: Removed LNI 192.168.202.135@tcp [13043.787703] Key type .llcrypt unregistered [13043.791327] Key type ._llcrypt unregistered [13068.965449] Key type ._llcrypt registered [13068.969641] Key type .llcrypt registered [13069.099207] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [13070.572355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13070.643547] alg: No test for adler32 (adler32-zlib) [13071.740740] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [13072.106522] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [13073.783320] Key type lgssc registered [13075.083284] Lustre: Echo OBD driver; http://www.lustre.org/ [13084.475260] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13084.505248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13086.272520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13091.556375] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13096.427209] LustreError: 428116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13096.445880] LustreError: 428116:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13101.539117] LustreError: 428117:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13103.352699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13104.147930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13110.849932] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13121.189498] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13128.444081] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13136.475238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13136.974580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13144.100604] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13152.022790] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13156.535261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13161.964373] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13161.984821] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13161.993699] Lustre: Skipped 1 previous similar message [13163.128534] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.35@tcp (stopping) [13167.097398] LustreError: 430503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13167.138399] LustreError: 430503:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13167.308416] Lustre: server umount lustre-MDT0001 complete [13172.194560] LustreError: 428110:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13172.222589] LustreError: 428110:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13177.316356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13177.319151] LustreError: 428845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13177.323401] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13177.323411] Lustre: Skipped 1 previous similar message [13177.325541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13179.742644] Lustre: server umount lustre-OST0000 complete [13183.694830] Lustre: server umount lustre-MDT0000 complete [13196.665293] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [13199.519461] Key type lgssc unregistered [13199.809714] LNet: 431805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13199.815845] LNetError: 431805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13199.838266] LNet: Removed LNI 192.168.202.135@tcp [13200.865493] Key type .llcrypt unregistered [13200.867140] Key type ._llcrypt unregistered [13226.891990] Key type ._llcrypt registered [13226.895490] Key type .llcrypt registered [13227.054423] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [13245.708903] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [13246.868480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13246.958146] alg: No test for adler32 (adler32-zlib) [13247.966034] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [13248.207633] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [13249.879224] Key type lgssc registered [13251.035234] Lustre: Echo OBD driver; http://www.lustre.org/ [13290.437140] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [13307.487812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13307.530387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13308.854961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13308.886815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13308.944608] Lustre: lustre-MDT0000: new disk, initializing [13309.027390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13309.043598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13314.067673] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13330.239604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13330.471188] Lustre: lustre-OST0000: new disk, initializing [13330.475454] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13330.479883] Lustre: Skipped 1 previous similar message [13330.484164] Lustre: 436837:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13330.557919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13337.754462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13337.776932] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13337.825139] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13338.328329] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13347.631528] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13363.673179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13363.774850] Lustre: 435819:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13363.853611] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13363.909590] Lustre: lustre-MDT0001: new disk, initializing [13364.005145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13364.075113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13364.086285] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13368.694641] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13378.948845] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13379.240967] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13384.987856] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13385.229415] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13390.953293] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.35@tcp (stopping) [13391.844140] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13391.859266] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13391.883336] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13391.893417] Lustre: Skipped 1 previous similar message [13396.040623] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.35@tcp (stopping) [13396.043549] Lustre: Skipped 1 previous similar message [13396.585410] Lustre: server umount lustre-MDT0001 complete [13402.096338] LustreError: 435825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13402.134428] LustreError: 435825:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13407.201175] LustreError: 435829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13407.243465] LustreError: 435829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13412.331107] LustreError: 435825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13412.366155] LustreError: 435825:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13414.882272] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13414.886433] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13414.894706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13417.444302] LustreError: 435829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13420.993295] Lustre: server umount lustre-OST0000 complete [13422.569723] LustreError: 435830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13431.929665] Lustre: server umount lustre-MDT0000 complete [13445.915509] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [13448.349804] Key type lgssc unregistered [13448.599579] LNet: 440587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13448.615226] LNetError: 440587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13448.634358] LNet: Removed LNI 192.168.202.135@tcp [13449.492990] Key type .llcrypt unregistered [13449.508811] Key type ._llcrypt unregistered [13486.552156] Key type ._llcrypt registered [13486.553762] Key type .llcrypt registered [13486.642130] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [13486.821398] Key type .llcrypt unregistered [13486.823749] Key type ._llcrypt unregistered [13502.837530] Key type ._llcrypt registered [13502.839638] Key type .llcrypt registered [13502.993180] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [13504.437658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13504.486933] alg: No test for adler32 (adler32-zlib) [13505.556669] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [13505.870327] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [13507.615176] Key type lgssc registered [13508.722924] Lustre: Echo OBD driver; http://www.lustre.org/ [13536.369158] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13536.448699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13537.851327] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13537.894911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13538.004427] Lustre: lustre-MDT0000: new disk, initializing [13538.194106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13538.238687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13543.858679] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13555.513936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13555.655971] Lustre: 443963:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13555.820733] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13555.825247] Lustre: Skipped 1 previous similar message [13555.944770] Lustre: lustre-MDT0001: new disk, initializing [13556.072630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13556.116041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13556.122959] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13560.803898] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13570.564093] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13577.541390] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13586.544993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13586.815775] Lustre: lustre-OST0000: new disk, initializing [13586.822848] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13586.841242] Lustre: 445931:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13586.996313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13595.458399] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13596.431180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13596.453618] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13596.678281] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13607.372703] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13617.121556] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13617.130307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13617.145534] Lustre: Skipped 1 previous similar message [13617.157027] Lustre: Skipped 2 previous similar messages [13621.451133] Lustre: server umount lustre-MDT0000 complete [13625.306270] LustreError: 443953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788481999 with bad export cookie 15591729661877747739 [13625.318162] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13625.331567] LustreError: 443953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13625.858849] Lustre: server umount lustre-MDT0001 complete [13635.490432] Lustre: 447114:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [13640.999114] Lustre: server umount lustre-OST0000 complete [13652.781280] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 20:33:45 (1788482025) [13661.971857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13662.341972] LustreError: 447964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13662.362542] LustreError: 447964:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13662.411863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13667.685855] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13667.813430] LustreError: 447965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13672.931489] LustreError: 447964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13676.088350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13676.451321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13681.138617] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13688.658249] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13696.874485] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13717.571273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13717.793945] Lustre: Found index 0 for lustre-OST0000, updating log [13718.089828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13725.352426] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13733.445678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13734.004403] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13741.540059] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13741.552400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13741.580950] Lustre: Skipped 1 previous similar message [13741.593318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13742.691352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13742.702850] Lustre: Skipped 2 previous similar messages [13745.787209] Lustre: server umount lustre-OST0000 complete [13763.554644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13763.560510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13763.570657] Lustre: Skipped 1 previous similar message [13763.575172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13769.106629] Lustre: server umount lustre-MDT0000 complete [13769.701722] LustreError: 447964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13769.741025] LustreError: 447964:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13773.481912] LustreError: 448699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788482147 with bad export cookie 15591729661877749160 [13773.487380] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13773.492388] LustreError: 448699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [13773.979913] Lustre: server umount lustre-MDT0001 complete [13782.432808] LNet: 1 local NIs in recovery (showing 1): 192.168.202.135@tcp [13786.552047] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 20:35:58 (1788482158) [14408.550780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14408.925072] LustreError: 454375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14408.949772] LustreError: 454375:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14409.051396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14414.026696] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [14414.308722] LustreError: 454376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14414.321459] LustreError: 454376:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14418.403066] LustreError: 454375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14422.644345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14423.083861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14427.775528] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [14435.608314] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14441.808128] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14460.383571] LNet: 1 local NIs in recovery (showing 1): 192.168.202.135@tcp [14462.260398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14462.505376] Lustre: Found index 0 for lustre-OST0000, updating log [14462.691387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14469.029786] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [14475.588297] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14481.325527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14481.334258] Lustre: Skipped 1 previous similar message [14481.341598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14481.346518] Lustre: Skipped 2 previous similar messages [14483.605634] Lustre: server umount lustre-OST0000 complete [14489.508220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14489.511810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14489.514526] Lustre: Skipped 1 previous similar message [14489.524649] Lustre: Skipped 1 previous similar message [14490.532962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14490.543789] Lustre: Skipped 1 previous similar message [14492.578624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14492.591517] Lustre: Skipped 3 previous similar messages [14494.903245] Lustre: server umount lustre-MDT0000 complete [14495.663114] LustreError: 454376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14495.690958] LustreError: 454376:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14499.021877] LustreError: 456228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788482872 with bad export cookie 15591729661877750273 [14499.041200] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14499.603161] Lustre: server umount lustre-MDT0001 complete [14520.863220] LNet: 1 local NIs in recovery (showing 1): 192.168.202.135@tcp [14529.347574] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [14542.340452] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [14581.252829] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 20:49:14 (1788482954) [14603.042767] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [14606.506282] Key type lgssc unregistered [14606.794455] LNet: 460987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14606.798115] LNetError: 460987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14722.542775] LNet: Removed LNI 192.168.202.135@tcp [14723.558814] Key type .llcrypt unregistered [14723.561163] Key type ._llcrypt unregistered [14745.153428] Key type ._llcrypt registered [14745.156699] Key type .llcrypt registered [14745.329783] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [14747.010858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14747.117093] alg: No test for adler32 (adler32-zlib) [14748.377879] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [14748.732311] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [14750.448486] Key type lgssc registered [14752.246948] Lustre: Echo OBD driver; http://www.lustre.org/ [14757.171766] LNet: Added LNI 192.168.202.135@tcp100 [8/256/0/180] [14758.016747] LNet: Added LNI 192.168.202.135@tcp101 [8/256/0/180] [14758.999587] LNet: Added LNI 192.168.202.135@tcp102 [8/256/0/180] [14759.729839] LNet: Added LNI 192.168.202.135@tcp103 [8/256/0/180] [14760.617674] LNet: Added LNI 192.168.202.135@tcp104 [8/256/0/180] [14761.453594] LNet: Added LNI 192.168.202.135@tcp105 [8/256/0/180] [14762.349851] LNet: Added LNI 192.168.202.135@tcp106 [8/256/0/180] [14763.468553] LNet: Added LNI 192.168.202.135@tcp107 [8/256/0/180] [14764.466046] LNet: Added LNI 192.168.202.135@tcp108 [8/256/0/180] [14765.351263] LNet: Added LNI 192.168.202.135@tcp109 [8/256/0/180] [14766.431486] LNet: Added LNI 192.168.202.135@tcp110 [8/256/0/180] [14767.262434] LNet: Added LNI 192.168.202.135@tcp111 [8/256/0/180] [14768.207823] LNet: Added LNI 192.168.202.135@tcp112 [8/256/0/180] [14768.970500] LNet: Added LNI 192.168.202.135@tcp113 [8/256/0/180] [14769.860932] LNet: Added LNI 192.168.202.135@tcp114 [8/256/0/180] [14770.786240] LNet: Added LNI 192.168.202.135@tcp115 [8/256/0/180] [14771.631849] LNet: Added LNI 192.168.202.135@tcp116 [8/256/0/180] [14772.383447] LNet: Added LNI 192.168.202.135@tcp117 [8/256/0/180] [14773.242456] LNet: Added LNI 192.168.202.135@tcp118 [8/256/0/180] [14774.316239] LNet: Added LNI 192.168.202.135@tcp119 [8/256/0/180] [14775.113635] LNet: Added LNI 192.168.202.135@tcp120 [8/256/0/180] [14775.893458] LNet: Added LNI 192.168.202.135@tcp121 [8/256/0/180] [14776.589395] LNet: Added LNI 192.168.202.135@tcp122 [8/256/0/180] [14777.230658] LNet: Added LNI 192.168.202.135@tcp123 [8/256/0/180] [14777.969525] LNet: Added LNI 192.168.202.135@tcp124 [8/256/0/180] [14779.100412] LNet: Added LNI 192.168.202.135@tcp125 [8/256/0/180] [14780.044895] LNet: Added LNI 192.168.202.135@tcp126 [8/256/0/180] [14780.910642] LNet: Added LNI 192.168.202.135@tcp127 [8/256/0/180] [14781.831550] LNet: Added LNI 192.168.202.135@tcp128 [8/256/0/180] [14782.913555] LNet: Added LNI 192.168.202.135@tcp129 [8/256/0/180] [14783.901308] LNet: Added LNI 192.168.202.135@tcp130 [8/256/0/180] [14784.813359] LNet: Added LNI 192.168.202.135@tcp131 [8/256/0/180] [14785.628460] LNet: Added LNI 192.168.202.135@tcp132 [8/256/0/180] [14786.628760] LNet: Added LNI 192.168.202.135@tcp133 [8/256/0/180] [14787.612384] LNet: Added LNI 192.168.202.135@tcp134 [8/256/0/180] [14788.470549] LNet: Added LNI 192.168.202.135@tcp135 [8/256/0/180] [14796.616203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14796.640543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14797.758395] Lustre: 464016:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.135@tcp: can't add NID 192.168.202.135@tcp130: rc = -75 [14797.877288] LNetError: 464037:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.135 [14797.901134] LNetError: Protocol error connecting to 192.168.202.135@tcp118 at host 192.168.202.135:988: is it running a compatible version of Lustre? [14797.912294] LNetError: Skipped 4 previous similar messages [14797.920289] LNet: 7 local NIs in recovery (showing 7): 192.168.202.135@tcp122, 192.168.202.135@tcp121, 192.168.202.135@tcp123, 192.168.202.135@tcp124, 192.168.202.135@tcp126, 192.168.202.135@tcp118, 192.168.202.135@tcp127 [14798.007920] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14798.054441] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14798.123678] Lustre: lustre-MDT0000: new disk, initializing [14798.190687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14798.202330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14798.944546] LNetError: 461473:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.135 [14798.961674] LNetError: 461473:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 12 previous similar messages [14798.971379] LNetError: Protocol error connecting to 192.168.202.135@tcp115 at host 192.168.202.135:988: is it running a compatible version of Lustre? [14798.989460] LNetError: Skipped 6 previous similar messages [14801.056561] LNetError: 461476:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.135 [14801.062493] LNetError: 461476:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [14801.083360] LNetError: Protocol error connecting to 192.168.202.135@tcp121 at host 192.168.202.135:988: is it running a compatible version of Lustre? [14801.096395] LNetError: Skipped 10 previous similar messages [14802.681630] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [14805.157238] LNetError: 464040:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.135 [14805.177596] LNetError: 464040:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14805.193974] LNetError: Protocol error connecting to 192.168.202.135@tcp121 at host 192.168.202.135:988: is it running a compatible version of Lustre? [14812.595076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14812.743579] Lustre: 464035:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14812.831580] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14812.836383] Lustre: Skipped 1 previous similar message [14812.904534] Lustre: lustre-MDT0001: new disk, initializing [14812.928549] Lustre: 464835:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.135@tcp: can't add NID 192.168.202.135@tcp130: rc = -75 [14812.936872] Lustre: 464835:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14813.053238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14813.093587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14813.115167] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14813.347956] LNetError: 464037:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.135 [14813.358320] LNetError: Protocol error connecting to 192.168.202.135@tcp121 at host 192.168.202.135:988: is it running a compatible version of Lustre? [14817.323326] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [14826.749307] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14829.732686] LNetError: 461474:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.135 [14829.742141] LNetError: Protocol error connecting to 192.168.202.135@tcp121 at host 192.168.202.135:988: is it running a compatible version of Lustre? [14830.855582] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14836.556455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14836.867669] Lustre: lustre-OST0000: new disk, initializing [14836.871556] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14836.876584] Lustre: 466007:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14836.943187] Lustre: 466012:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.135@tcp: can't add NID 192.168.202.135@tcp130: rc = -75 [14836.948962] Lustre: 466012:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [14836.980526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14842.591275] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [14844.619894] Lustre: 466463:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.135@tcp: can't add NID 192.168.202.135@tcp130: rc = -75 [14844.628607] Lustre: 466463:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14844.713079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14844.742844] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14844.823463] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14847.076952] LNetError: 464042:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.135 [14847.101901] LNetError: 464042:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [14847.114592] LNetError: Protocol error connecting to 192.168.202.135@tcp117 at host 192.168.202.135:988: is it running a compatible version of Lustre? [14847.131371] LNetError: Skipped 9 previous similar messages [14852.887693] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14857.375157] LNet: 1 local NIs in recovery (showing 1): 192.168.202.135@tcp121 [14860.262704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14860.266071] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14860.280975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14860.283515] LustreError: Skipped 1 previous similar message [14860.299352] Lustre: Skipped 1 previous similar message [14865.376350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14865.381801] Lustre: Skipped 1 previous similar message [14866.550919] Lustre: server umount lustre-OST0000 complete [14874.596436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14874.605227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14874.617985] Lustre: Skipped 1 previous similar message [14874.638839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14875.620636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14879.841135] LNetError: 464042:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.135 [14879.850885] LNetError: 464042:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 17 previous similar messages [14879.865189] LNetError: Protocol error connecting to 192.168.202.135@tcp126 at host 192.168.202.135:988: is it running a compatible version of Lustre? [14879.872679] LNetError: Skipped 17 previous similar messages [14880.736777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14880.745509] Lustre: Skipped 3 previous similar messages [14885.343176] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14885.689097] Lustre: server umount lustre-MDT0000 complete [14890.982698] LustreError: 466006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788483264 with bad export cookie 4444074314205624793 [14890.985510] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14890.989454] LustreError: 466006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14891.363907] Lustre: server umount lustre-MDT0001 complete [14904.707536] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [14908.380735] Key type lgssc unregistered [14908.887958] LNet: 467921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14908.900763] LNetError: 467921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14908.907897] LNet: Removed LNI 192.168.202.135@tcp [14909.920860] LNet: Removed LNI 192.168.202.135@tcp100 [14909.948454] LNetError: 467921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14909.971633] LNetError: 467921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14909.995702] LNet: Removed LNI 192.168.202.135@tcp101 [14911.007158] LNet: Removed LNI 192.168.202.135@tcp102 [14911.010512] LNetError: 467921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14911.022306] LNetError: 467921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14911.029388] LNet: Removed LNI 192.168.202.135@tcp103 [14912.095377] LNet: Removed LNI 192.168.202.135@tcp104 [14913.119115] LNet: Removed LNI 192.168.202.135@tcp105 [14913.128197] LNetError: 467921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14913.132859] LNetError: 467921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14913.138529] LNet: Removed LNI 192.168.202.135@tcp106 [14914.143236] LNet: Removed LNI 192.168.202.135@tcp107 [14915.168950] LNet: Removed LNI 192.168.202.135@tcp108 [14916.191188] LNet: Removed LNI 192.168.202.135@tcp109 [14917.215432] LNet: Removed LNI 192.168.202.135@tcp110 [14917.218209] LNetError: 467921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14917.227493] LNetError: 467921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14917.234995] LNet: Removed LNI 192.168.202.135@tcp111 [14918.239656] LNet: Removed LNI 192.168.202.135@tcp112 [14919.263166] LNet: Removed LNI 192.168.202.135@tcp113 [14920.289585] LNet: Removed LNI 192.168.202.135@tcp114 [14921.311176] LNet: Removed LNI 192.168.202.135@tcp115 [14922.335222] LNet: Removed LNI 192.168.202.135@tcp116 [14923.359156] LNet: Removed LNI 192.168.202.135@tcp117 [14924.383217] LNet: Removed LNI 192.168.202.135@tcp118 [14925.407451] LNet: Removed LNI 192.168.202.135@tcp119 [14925.411902] LNetError: 467921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14925.422254] LNetError: 467921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14925.433856] LNet: Removed LNI 192.168.202.135@tcp120 [14925.438467] LNet: Removed LNI 192.168.202.135@tcp121 [14926.495355] LNet: Removed LNI 192.168.202.135@tcp122 [14927.524602] LNet: Removed LNI 192.168.202.135@tcp123 [14928.543209] LNet: Removed LNI 192.168.202.135@tcp124 [14929.567307] LNet: Removed LNI 192.168.202.135@tcp125 [14930.591449] LNet: Removed LNI 192.168.202.135@tcp126 [14931.615416] LNet: Removed LNI 192.168.202.135@tcp127 [14932.639808] LNet: Removed LNI 192.168.202.135@tcp128 [14933.666067] LNet: Removed LNI 192.168.202.135@tcp129 [14933.669479] LNet: Removed LNI 192.168.202.135@tcp130 [14933.671382] LNet: Removed LNI 192.168.202.135@tcp131 [14933.680407] LNet: Removed LNI 192.168.202.135@tcp132 [14933.690434] LNet: Removed LNI 192.168.202.135@tcp133 [14933.703352] LNet: Removed LNI 192.168.202.135@tcp134 [14933.721995] LNet: Removed LNI 192.168.202.135@tcp135 [14934.900977] Key type .llcrypt unregistered [14934.904430] Key type ._llcrypt unregistered [14958.060927] Key type ._llcrypt registered [14958.063255] Key type .llcrypt registered [14958.250638] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [14974.956777] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [14976.447885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14976.557917] alg: No test for adler32 (adler32-zlib) [14977.697530] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [14977.965814] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [14979.631130] Key type lgssc registered [14980.953560] Lustre: Echo OBD driver; http://www.lustre.org/ [15011.329436] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 20:56:24 (1788483384) [15031.746222] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [15035.672809] Key type lgssc unregistered [15035.979943] LNet: 472004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15035.990776] LNetError: 472004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15036.001763] LNet: Removed LNI 192.168.202.135@tcp [15037.201114] Key type .llcrypt unregistered [15037.202801] Key type ._llcrypt unregistered [15058.496181] Key type ._llcrypt registered [15058.498113] Key type .llcrypt registered [15058.632932] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [15059.874877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15060.002170] alg: No test for adler32 (adler32-zlib) [15061.029388] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [15061.232164] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [15062.887165] Key type lgssc registered [15064.220852] Lustre: Echo OBD driver; http://www.lustre.org/ [15072.717718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15072.752142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15074.043494] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15074.108620] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15074.233953] Lustre: lustre-MDT0000: new disk, initializing [15074.369118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15074.463899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15078.388220] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [15087.212565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15087.331698] Lustre: 473215:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15087.419547] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15087.424162] Lustre: Skipped 1 previous similar message [15087.477392] Lustre: lustre-MDT0001: new disk, initializing [15087.545878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15087.575079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15087.589526] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15092.447344] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [15102.612944] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15106.754448] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15112.165630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15112.564507] Lustre: lustre-OST0000: new disk, initializing [15112.573729] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15112.580729] Lustre: 475184:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15112.663992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15117.981560] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [15121.008442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15121.021777] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15121.092964] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15126.413145] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15129.904407] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing lnet_if_list [15135.658181] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15135.688358] LNet: Added LNI 192.168.202.135@tcp10 [8/256/0/180] [15142.542465] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15142.599455] LNet: Added LNI 192.168.202.135@tcp11 [8/256/0/180] [15147.336662] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15147.371947] LNet: Added LNI 192.168.202.135@tcp12 [8/256/0/180] [15152.398212] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15152.429512] LNet: Added LNI 192.168.202.135@tcp13 [8/256/0/180] [15156.916253] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15156.940599] LNet: Added LNI 192.168.202.135@tcp14 [8/256/0/180] [15161.442828] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15161.465923] LNet: Added LNI 192.168.202.135@tcp15 [8/256/0/180] [15167.295310] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15167.320494] LNet: Added LNI 192.168.202.135@tcp16 [8/256/0/180] [15172.277951] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15172.311965] LNet: Added LNI 192.168.202.135@tcp17 [8/256/0/180] [15178.553628] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15178.582414] LNet: Added LNI 192.168.202.135@tcp18 [8/256/0/180] [15183.970294] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15184.017900] LNet: Added LNI 192.168.202.135@tcp19 [8/256/0/180] [15189.715979] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15189.740782] LNet: Added LNI 192.168.202.135@tcp20 [8/256/0/180] [15197.569500] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15197.609799] LNet: Added LNI 192.168.202.135@tcp21 [8/256/0/180] [15204.249825] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15204.302656] LNet: Added LNI 192.168.202.135@tcp22 [8/256/0/180] [15211.604625] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15211.640705] LNet: Added LNI 192.168.202.135@tcp23 [8/256/0/180] [15217.572304] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15217.664249] LNet: Added LNI 192.168.202.135@tcp24 [8/256/0/180] [15223.955156] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15223.979570] LNet: Added LNI 192.168.202.135@tcp25 [8/256/0/180] [15229.311726] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15229.348656] LNet: Added LNI 192.168.202.135@tcp26 [8/256/0/180] [15235.222534] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15235.248344] LNet: Added LNI 192.168.202.135@tcp27 [8/256/0/180] [15241.651289] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15241.674268] LNet: Added LNI 192.168.202.135@tcp28 [8/256/0/180] [15247.672961] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15247.694401] LNet: Added LNI 192.168.202.135@tcp29 [8/256/0/180] [15253.508411] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15253.534117] LNet: Added LNI 192.168.202.135@tcp30 [8/256/0/180] [15259.822156] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15259.849952] LNet: Added LNI 192.168.202.135@tcp31 [8/256/0/180] [15266.633445] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15266.667166] LNet: Added LNI 192.168.202.135@tcp32 [8/256/0/180] [15272.372182] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15272.418884] LNet: Added LNI 192.168.202.135@tcp33 [8/256/0/180] [15279.005510] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15279.029790] LNet: Added LNI 192.168.202.135@tcp34 [8/256/0/180] [15286.543702] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15286.570471] LNet: Added LNI 192.168.202.135@tcp35 [8/256/0/180] [15294.514049] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15294.591767] LNet: Added LNI 192.168.202.135@tcp36 [8/256/0/180] [15303.067867] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15303.127377] LNet: Added LNI 192.168.202.135@tcp37 [8/256/0/180] [15310.372326] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15310.396688] LNet: Added LNI 192.168.202.135@tcp38 [8/256/0/180] [15317.946337] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15317.977305] LNet: Added LNI 192.168.202.135@tcp39 [8/256/0/180] [15323.973934] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15324.001263] LNet: Added LNI 192.168.202.135@tcp40 [8/256/0/180] [15330.408500] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15330.434618] LNet: Added LNI 192.168.202.135@tcp41 [8/256/0/180] [15338.065271] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15338.092093] LNet: Added LNI 192.168.202.135@tcp42 [8/256/0/180] [15344.850404] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15344.900046] LNet: Added LNI 192.168.202.135@tcp43 [8/256/0/180] [15353.961061] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15354.026167] LNet: Added LNI 192.168.202.135@tcp44 [8/256/0/180] [15363.497245] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15363.528681] LNet: Added LNI 192.168.202.135@tcp45 [8/256/0/180] [15372.717828] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15372.753553] LNet: Added LNI 192.168.202.135@tcp46 [8/256/0/180] [15381.777136] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15381.806350] LNet: Added LNI 192.168.202.135@tcp47 [8/256/0/180] [15390.575884] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15390.599723] LNet: Added LNI 192.168.202.135@tcp48 [8/256/0/180] [15400.507150] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15400.536489] LNet: Added LNI 192.168.202.135@tcp49 [8/256/0/180] [15408.978612] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15409.010932] LNet: Added LNI 192.168.202.135@tcp50 [8/256/0/180] [15428.577229] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15428.586360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15428.607460] Lustre: Skipped 2 previous similar messages [15428.621414] Lustre: Skipped 2 previous similar messages [15433.710335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15433.717543] Lustre: Skipped 2 previous similar messages [15433.858290] Lustre: server umount lustre-MDT0000 complete [15437.085600] LustreError: 498208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788483810 with bad export cookie 7114196744719253094 [15437.091284] LustreError: 498208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15437.094490] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15437.376490] Lustre: server umount lustre-MDT0001 complete [15447.968636] Lustre: 498659:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [15453.154441] Lustre: 498662:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [15454.179297] Lustre: 472485:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788483812/real 1788483812] req@00000000cc2bfc59 x1875360806513920/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788483828 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15454.212288] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15454.226232] Lustre: 498659:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [15454.753113] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [15458.335892] Lustre: 498662:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [15458.337845] Lustre: 472485:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788483816/real 1788483816] req@000000001fb2e7a3 x1875360806514048/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788483832 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15462.773911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15463.098272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15464.162505] Lustre: 472487:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788483821/real 1788483821] req@00000000df234c46 x1875360806514432/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788483837 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15464.199421] Lustre: 472487:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [15467.540477] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [15469.151643] Lustre: 472484:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788483827/real 1788483827] req@0000000007efe02a x1875360806514688/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788483843 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15470.561708] LustreError: 475177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15470.573732] LustreError: 475177:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [15474.913463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15475.207138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15479.195744] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [15480.314427] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15483.437288] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15487.046380] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15494.458951] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15514.083356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15514.092694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15514.097801] Lustre: Skipped 1 previous similar message [15519.201886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15520.509860] Lustre: server umount lustre-OST0000 complete [15526.368545] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15526.374267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15526.389505] Lustre: Skipped 1 previous similar message [15526.401403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15526.416572] Lustre: Skipped 1 previous similar message [15531.738949] Lustre: server umount lustre-MDT0000 complete [15534.562187] LustreError: 499438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15534.595422] LustreError: 499438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15536.254854] LustreError: 473208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788483910 with bad export cookie 7114196744719255768 [15536.257044] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15536.270890] LustreError: 473208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15536.576687] Lustre: server umount lustre-MDT0001 complete [15548.742657] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [15551.315325] Key type lgssc unregistered [15551.710557] LNet: 502234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15551.718071] LNetError: 502234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15551.723610] LNet: Removed LNI 192.168.202.135@tcp [15551.726529] LNet: Removed LNI 192.168.202.135@tcp10 [15551.730689] LNet: Removed LNI 192.168.202.135@tcp11 [15551.734706] LNet: Removed LNI 192.168.202.135@tcp12 [15551.737283] LNet: Removed LNI 192.168.202.135@tcp13 [15551.739749] LNet: Removed LNI 192.168.202.135@tcp14 [15551.742357] LNet: Removed LNI 192.168.202.135@tcp15 [15551.745976] LNet: Removed LNI 192.168.202.135@tcp16 [15551.748281] LNet: Removed LNI 192.168.202.135@tcp17 [15551.750452] LNet: Removed LNI 192.168.202.135@tcp18 [15551.753103] LNet: Removed LNI 192.168.202.135@tcp19 [15551.755655] LNet: Removed LNI 192.168.202.135@tcp20 [15551.758476] LNet: Removed LNI 192.168.202.135@tcp21 [15551.762095] LNet: Removed LNI 192.168.202.135@tcp22 [15551.765992] LNet: Removed LNI 192.168.202.135@tcp23 [15551.768823] LNet: Removed LNI 192.168.202.135@tcp24 [15551.772786] LNet: Removed LNI 192.168.202.135@tcp25 [15551.782770] LNet: Removed LNI 192.168.202.135@tcp26 [15551.787629] LNet: Removed LNI 192.168.202.135@tcp27 [15551.791436] LNet: Removed LNI 192.168.202.135@tcp28 [15551.798691] LNet: Removed LNI 192.168.202.135@tcp29 [15551.804645] LNet: Removed LNI 192.168.202.135@tcp30 [15551.810243] LNet: Removed LNI 192.168.202.135@tcp31 [15551.820699] LNet: Removed LNI 192.168.202.135@tcp32 [15551.826313] LNet: Removed LNI 192.168.202.135@tcp33 [15551.832076] LNet: Removed LNI 192.168.202.135@tcp34 [15551.835334] LNet: Removed LNI 192.168.202.135@tcp35 [15551.837790] LNet: Removed LNI 192.168.202.135@tcp36 [15551.841672] LNet: Removed LNI 192.168.202.135@tcp37 [15551.851675] LNet: Removed LNI 192.168.202.135@tcp38 [15551.859712] LNet: Removed LNI 192.168.202.135@tcp39 [15551.866195] LNet: Removed LNI 192.168.202.135@tcp40 [15551.870445] LNet: Removed LNI 192.168.202.135@tcp41 [15551.876275] LNet: Removed LNI 192.168.202.135@tcp42 [15551.884284] LNet: Removed LNI 192.168.202.135@tcp43 [15551.897051] LNet: Removed LNI 192.168.202.135@tcp44 [15551.900717] LNet: Removed LNI 192.168.202.135@tcp45 [15551.902584] LNet: Removed LNI 192.168.202.135@tcp46 [15551.906249] LNet: Removed LNI 192.168.202.135@tcp47 [15551.914031] LNet: Removed LNI 192.168.202.135@tcp48 [15551.917462] LNet: Removed LNI 192.168.202.135@tcp49 [15551.929674] LNet: Removed LNI 192.168.202.135@tcp50 [15552.892464] Key type .llcrypt unregistered [15552.894780] Key type ._llcrypt unregistered [15576.529557] Key type ._llcrypt registered [15576.532250] Key type .llcrypt registered [15576.599466] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [15592.276816] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [15593.299981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15593.427268] alg: No test for adler32 (adler32-zlib) [15594.503564] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [15594.743602] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [15596.464138] Key type lgssc registered [15597.774965] Lustre: Echo OBD driver; http://www.lustre.org/ [15633.365069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15633.420592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15634.808534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15634.847680] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15634.974110] Lustre: lustre-MDT0000: new disk, initializing [15635.213204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15635.280703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15641.043358] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [15656.252034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15656.452952] Lustre: 505426:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15656.638836] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15656.643122] Lustre: Skipped 1 previous similar message [15656.823288] Lustre: lustre-MDT0001: new disk, initializing [15657.011633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15657.106235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15657.141104] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15662.030870] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [15671.812182] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15678.052182] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15686.445541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15686.609842] Lustre: *** cfs_fail_loc=90f, val=0*** [15686.791397] Lustre: lustre-OST0000: new disk, initializing [15686.799815] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15686.815593] Lustre: 507445:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15686.972916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15694.545818] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [15694.568988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15694.581706] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15694.673952] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15703.321966] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15710.180267] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15710.181822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15710.186333] LustreError: Skipped 1 previous similar message [15710.198878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15715.305892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15715.310462] Lustre: Skipped 2 previous similar messages [15716.163774] Lustre: server umount lustre-OST0000 complete [15720.417997] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15720.433141] Lustre: Skipped 1 previous similar message [15720.437566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15725.536584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15725.550290] Lustre: Skipped 2 previous similar messages [15725.833660] Lustre: server umount lustre-MDT0000 complete [15729.206069] LustreError: 505418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788484103 with bad export cookie 14638710337682353835 [15729.210673] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15729.215048] LustreError: 505418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15729.787162] Lustre: server umount lustre-MDT0001 complete [15743.211879] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [15746.530686] Key type lgssc unregistered [15746.797365] LNet: 509351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15746.810905] LNetError: 509351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15746.839314] LNet: Removed LNI 192.168.202.135@tcp [15747.736203] Key type .llcrypt unregistered [15747.739557] Key type ._llcrypt unregistered [15774.180509] Key type ._llcrypt registered [15774.183568] Key type .llcrypt registered [15774.365299] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [15776.464666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15776.591237] alg: No test for adler32 (adler32-zlib) [15778.180660] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [15778.544710] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [15780.295555] Key type lgssc registered [15781.544551] Lustre: Echo OBD driver; http://www.lustre.org/ [15789.836093] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15789.857441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15791.433928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15796.887968] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [15801.827363] LustreError: 510620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15801.867132] LustreError: 510620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15806.769833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15806.948469] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [15807.080564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15812.364829] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [15821.136938] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15827.375731] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15834.582583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15834.922576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15842.390102] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [15851.095240] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15861.979594] loop0: detected capacity change from 0 to 204800000 [15862.224419] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15862.235724] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15862.246844] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15868.490255] loop0: detected capacity change from 0 to 204800000 [15868.734459] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15868.753528] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15868.765866] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15875.339886] loop0: detected capacity change from 0 to 204800000 [15881.607342] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15881.759593] Lustre: fs15246-OST0000: new disk, initializing [15881.763478] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15881.769538] Lustre: 514214:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15881.832236] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15889.075128] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [15899.584820] loop1: detected capacity change from 0 to 204800000 [15905.446648] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15905.521490] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15905.602360] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15905.675928] Lustre: fs15246-MDT0000: new disk, initializing [15905.891136] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15905.938515] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15906.039739] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15907.106957] Lustre: 515335:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000001c7295ac x1875361558216832/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788484297 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15910.299679] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [15916.198101] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15916.231512] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15916.284601] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15940.584234] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15940.636287] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15944.722150] Lustre: server umount fs15246-MDT0000 complete [15945.699198] LustreError: 510615:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15950.824920] LustreError: 511905:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15955.222594] Lustre: server umount fs15246-OST0000 complete [15969.250187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15969.262431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15969.277684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15970.277513] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15970.285882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15971.301216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15971.313795] Lustre: Skipped 2 previous similar messages [15974.750576] Lustre: server umount lustre-OST0000 complete [15981.026655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15981.038778] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15981.064549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15984.589806] Lustre: server umount lustre-MDT0000 complete [15986.664888] LustreError: 510620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15986.681367] LustreError: 510620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15988.064238] LustreError: 510602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788484361 with bad export cookie 1154677143787518546 [15988.076696] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15988.386770] Lustre: server umount lustre-MDT0001 complete [16002.621536] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [16008.671554] Key type lgssc unregistered [16009.580326] LNet: 518435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16009.603954] LNetError: 518435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16009.685402] LNet: Removed LNI 192.168.202.135@tcp [16011.870285] Key type .llcrypt unregistered [16011.877755] Key type ._llcrypt unregistered [16033.693571] Key type ._llcrypt registered [16033.695525] Key type .llcrypt registered [16033.785193] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [16034.967868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16035.199986] alg: No test for adler32 (adler32-zlib) [16036.335845] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [16036.524914] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [16038.183416] Key type lgssc registered [16039.632668] Lustre: Echo OBD driver; http://www.lustre.org/ [16083.503343] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [16094.589411] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [16140.674524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16140.702831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16142.103259] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16142.175403] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16142.291885] Lustre: lustre-MDT0000: new disk, initializing [16142.434167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16142.487740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16147.514059] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [16157.064546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16157.175493] Lustre: 522902:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16157.258969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16157.262291] Lustre: Skipped 1 previous similar message [16157.373972] Lustre: lustre-MDT0001: new disk, initializing [16157.477229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16157.530944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16157.562229] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16162.735590] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [16171.603752] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16177.820467] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16185.050383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16185.448956] Lustre: lustre-OST0000: new disk, initializing [16185.456847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16185.485636] Lustre: 524868:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16185.665223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16193.406554] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [16195.204441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16195.209879] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16195.464384] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16204.610929] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16210.101380] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16210.352815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16214.587555] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16214.830133] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16221.153265] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16221.164208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16221.164574] LustreError: Skipped 1 previous similar message [16221.187840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16221.194598] Lustre: Skipped 1 previous similar message [16226.277429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16226.292155] Lustre: Skipped 1 previous similar message [16230.879510] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16231.204620] Lustre: server umount lustre-OST0000 complete [16239.594496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16239.625989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16239.667682] Lustre: Skipped 1 previous similar message [16239.690276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16241.634847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16243.994310] Lustre: server umount lustre-MDT0000 complete [16248.911799] LustreError: 522893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788484622 with bad export cookie 7804376870169899199 [16248.926946] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16249.364481] Lustre: server umount lustre-MDT0001 complete [16279.020443] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [16292.616608] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [16333.497623] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16023 sec ========= 21:18:26 (1788484706) [16334.942354] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:18:27 (1788484707) === [16337.958721] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:18:30 (1788484710) === [16378.469487] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [16380.646690] Key type lgssc unregistered [16380.902589] LNet: 531516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16380.909688] LNetError: 531516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16380.938144] LNet: Removed LNI 192.168.202.135@tcp [16382.073747] Key type .llcrypt unregistered [16382.081773] Key type ._llcrypt unregistered