[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 863491644 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002367] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004023] kvm-guest: setup PV IPIs [ 0.006980] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009122] LSM: Security Framework initializing [ 0.010052] Yama: becoming mindful. [ 0.011026] SELinux: Initializing. [ 0.012047] *** VALIDATE selinux *** [ 0.020672] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025332] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026120] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027079] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028084] *** VALIDATE tmpfs *** [ 0.030064] *** VALIDATE proc *** [ 0.031192] *** VALIDATE cgroup *** [ 0.032007] *** VALIDATE cgroup2 *** [ 0.033224] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034214] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035019] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036035] Spectre V2 : User space: Vulnerable [ 0.037019] Speculative Store Bypass: Vulnerable [ 0.040312] debug: unmapping init [mem 0xffffffff99259000-0xffffffff99260fff] [ 0.043000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043880] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044030] ... version: 2 [ 0.045008] ... bit width: 48 [ 0.046007] ... generic registers: 4 [ 0.047009] ... value mask: 0000ffffffffffff [ 0.048010] ... max period: 00007fffffffffff [ 0.049009] ... fixed-purpose events: 3 [ 0.050007] ... event mask: 000000070000000f [ 0.051276] rcu: Hierarchical SRCU implementation. [ 0.053522] smp: Bringing up secondary CPUs ... [ 0.054514] x86: Booting SMP configuration: [ 0.055032] .... node #0, CPUs: #1 #2 #3 [ 0.062647] smp: Brought up 1 node, 4 CPUs [ 0.064025] smpboot: Max logical packages: 1 [ 0.065013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.097066] node 0 deferred pages initialised in 30ms [ 0.101033] devtmpfs: initialized [ 0.102239] x86/mm: Memory block size: 128MB [ 0.104739] gcov: version magic: 0x41383552 [ 0.106131] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.107057] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.108198] pinctrl core: initialized pinctrl subsystem [ 0.109124] [ 0.109597] ************************************************************* [ 0.110006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.111007] ** ** [ 0.112021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.113007] ** ** [ 0.114032] ** This means that this kernel is built to expose internal ** [ 0.115006] ** IOMMU data structures, which may compromise security on ** [ 0.116020] ** your system. ** [ 0.117007] ** ** [ 0.118011] ** If you see this message and you are not debugging the ** [ 0.119021] ** kernel, report this immediately to your vendor! ** [ 0.120006] ** ** [ 0.121120] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.122008] ************************************************************* [ 0.123662] NET: Registered protocol family 16 [ 0.124405] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.125064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.126054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.127654] cpuidle: using governor menu [ 0.129096] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.131561] PCI: Using configuration type 1 for base access [ 0.134132] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.143124] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.144052] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.145271] cryptd: max_cpu_qlen set to 1000 [ 0.147103] ACPI: Added _OSI(Module Device) [ 0.148008] ACPI: Added _OSI(Processor Device) [ 0.149007] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.150015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.154076] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.156811] ACPI: Interpreter enabled [ 0.157047] ACPI: PM: (supports S0 S3 S4 S5) [ 0.158028] ACPI: Using IOAPIC for interrupt routing [ 0.159112] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.160410] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.171054] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.172108] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.173012] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.174148] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.176289] acpiphp: Slot [2] registered [ 0.177219] acpiphp: Slot [5] registered [ 0.178071] acpiphp: Slot [6] registered [ 0.179098] acpiphp: Slot [7] registered [ 0.180078] acpiphp: Slot [8] registered [ 0.181081] acpiphp: Slot [9] registered [ 0.182117] acpiphp: Slot [10] registered [ 0.183151] acpiphp: Slot [3] registered [ 0.184083] acpiphp: Slot [4] registered [ 0.185054] acpiphp: Slot [11] registered [ 0.186107] acpiphp: Slot [12] registered [ 0.187054] acpiphp: Slot [13] registered [ 0.188068] acpiphp: Slot [14] registered [ 0.189053] acpiphp: Slot [15] registered [ 0.190212] acpiphp: Slot [16] registered [ 0.191065] acpiphp: Slot [17] registered [ 0.192049] acpiphp: Slot [18] registered [ 0.193077] acpiphp: Slot [19] registered [ 0.194066] acpiphp: Slot [20] registered [ 0.195207] acpiphp: Slot [21] registered [ 0.196125] acpiphp: Slot [22] registered [ 0.197125] acpiphp: Slot [23] registered [ 0.198057] acpiphp: Slot [24] registered [ 0.199053] acpiphp: Slot [25] registered [ 0.200063] acpiphp: Slot [26] registered [ 0.201068] acpiphp: Slot [27] registered [ 0.202073] acpiphp: Slot [28] registered [ 0.203073] acpiphp: Slot [29] registered [ 0.204104] acpiphp: Slot [30] registered [ 0.205056] acpiphp: Slot [31] registered [ 0.206099] PCI host bridge to bus 0000:00 [ 0.207025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.209023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.212018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.215015] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.217026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.219026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.221249] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.225171] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.228152] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.238033] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.241888] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.245013] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.248012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.250014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.253313] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.255954] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.258041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.260774] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.267010] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.284013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.294013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.303138] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.328019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.342013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.515000] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.515000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.515000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.515000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.515000] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.515000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.515000] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.515000] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.515000] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.515000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.525012] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.534009] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.566013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.634000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.646011] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.921011] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.948013] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.964397] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.973011] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.983012] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.004011] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.029973] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.032344] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.038364] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.040257] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.043197] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.052151] iommu: Default domain type: Passthrough [ 1.055164] SCSI subsystem initialized [ 1.057078] ACPI: bus type USB registered [ 1.059080] usbcore: registered new interface driver usbfs [ 1.061049] usbcore: registered new interface driver hub [ 1.064058] usbcore: registered new device driver usb [ 1.067516] pps_core: LinuxPPS API ver. 1 registered [ 1.071010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.075082] PTP clock support registered [ 1.077238] EDAC MC: Ver: 3.0.0 [ 1.078663] PCI: Using ACPI for IRQ routing [ 1.079778] NetLabel: Initializing [ 1.082007] NetLabel: domain hash size = 128 [ 1.083005] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.085059] NetLabel: unlabeled traffic allowed by default [ 1.086320] vgaarb: loaded [ 1.088084] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.089007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.099230] clocksource: Switched to clocksource kvm-clock [ 1.207802] VFS: Disk quotas dquot_6.6.0 [ 1.209396] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.211943] *** VALIDATE ramfs *** [ 1.213152] *** VALIDATE hugetlbfs *** [ 1.215201] pnp: PnP ACPI init [ 1.217563] pnp: PnP ACPI: found 6 devices [ 1.233798] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.236742] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.239247] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.241320] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.243781] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.246201] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.249033] NET: Registered protocol family 2 [ 1.251235] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.255784] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.259379] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.264395] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.267746] TCP: Hash tables configured (established 65536 bind 65536) [ 1.270646] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.273896] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.276667] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.280621] NET: Registered protocol family 1 [ 1.283925] RPC: Registered named UNIX socket transport module. [ 1.286801] RPC: Registered udp transport module. [ 1.288887] RPC: Registered tcp transport module. [ 1.290532] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.292682] NET: Registered protocol family 44 [ 1.294136] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.296525] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.298520] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.300641] PCI: CLS 0 bytes, default 64 [ 1.302158] Unpacking initramfs... [ 2.965839] debug: unmapping init [mem 0xffff92443cc54000-0xffff92443ffbffff] [ 2.970200] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.972477] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.975045] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.523406] Initialise system trusted keyrings [ 3.525435] Key type blacklist registered [ 3.527428] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.537431] zbud: loaded [ 3.540245] *** VALIDATE nfs *** [ 3.541224] *** VALIDATE nfs4 *** [ 3.542771] pstore: using deflate compression [ 3.571854] Platform Keyring initialized [ 4.026784] NET: Registered protocol family 38 [ 4.030655] Key type asymmetric registered [ 4.034356] Asymmetric key parser 'x509' registered [ 4.036076] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.039192] io scheduler mq-deadline registered [ 4.040875] io scheduler kyber registered [ 4.042715] io scheduler bfq registered [ 4.045613] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.048691] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.051582] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.056729] ACPI: Power Button [PWRF] [ 4.061885] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.068977] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.224442] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.240203] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.303893] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.345651] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.381542] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.392058] Non-volatile memory driver v1.3 [ 4.396543] Linux agpgart interface v0.103 [ 4.880630] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 4.886181] vda: detected capacity change from 0 to 74702848 [ 5.086786] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.089431] vdb: detected capacity change from 0 to 1073741824 [ 5.101598] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.104212] vdc: detected capacity change from 0 to 2621440000 [ 5.117540] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.120312] vdd: detected capacity change from 0 to 2621440000 [ 5.137442] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.144061] vde: detected capacity change from 0 to 4294967296 [ 5.171892] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.179136] vdf: detected capacity change from 0 to 4294967296 [ 5.190550] libphy: Fixed MDIO Bus: probed [ 5.201661] usbcore: registered new interface driver usbserial_generic [ 5.208244] usbserial: USB Serial support registered for generic [ 5.212510] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.222689] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.227380] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.232295] mousedev: PS/2 mouse device common for all mice [ 5.237483] rtc_cmos 00:05: RTC can wake from S4 [ 5.247229] rtc_cmos 00:05: registered as rtc0 [ 5.251453] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.259558] intel_pstate: CPU model not supported [ 5.268522] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.276499] hid: raw HID events driver (C) Jiri Kosina [ 5.284678] usbcore: registered new interface driver usbhid [ 5.290439] usbhid: USB HID core driver [ 5.294440] drop_monitor: Initializing network drop monitor service [ 5.301458] Initializing XFRM netlink socket [ 5.305873] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.315643] NET: Registered protocol family 10 [ 5.322817] Segment Routing with IPv6 [ 5.325658] NET: Registered protocol family 17 [ 5.334050] mpls_gso: MPLS GSO support [ 5.346453] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.370663] RAS: Correctable Errors collector initialized. [ 5.376087] AVX version of gcm_enc/dec engaged. [ 5.381069] AES CTR mode by8 optimization enabled [ 5.647192] sched_clock: Marking stable (5647082298, 0)->(6791394466, -1144312168) [ 5.651834] registered taskstats version 1 [ 5.654676] Loading compiled-in X.509 certificates [ 5.659327] zswap: loaded using pool lzo/zbud [ 5.689627] Key type big_key registered [ 5.703562] Key type encrypted registered [ 5.705561] ima: No TPM chip found, activating TPM-bypass! [ 5.708312] ima: Allocated hash algorithm: sha1 [ 5.768664] ima: No architecture policies found [ 5.778753] evm: Initialising EVM extended attributes: [ 5.789419] evm: security.selinux [ 5.793189] evm: security.ima [ 5.794878] evm: security.capability [ 5.798447] evm: HMAC attrs: 0x1 [ 5.811245] rtc_cmos 00:05: setting system clock to 2026-07-30 05:11:18 UTC (1785388278) [ 5.821634] debug: unmapping init [mem 0xffffffff9a203000-0xffffffff9a3fffff] [ 5.826655] debug: unmapping init [mem 0xffffffff98f82000-0xffffffff99258fff] [ 5.834332] Write protecting the kernel read-only data: 28672k [ 5.840925] debug: unmapping init [mem 0xffffffff97603000-0xffffffff977fffff] [ 5.846674] debug: unmapping init [mem 0xffffffff97f14000-0xffffffff97ffffff] [ 5.897902] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.909531] systemd[1]: Detected virtualization kvm. [ 5.912097] systemd[1]: Detected architecture x86-64. [ 5.914492] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.937886] systemd[1]: No hostname configured. [ 5.940176] systemd[1]: Set hostname to . [ 5.943076] random: systemd: uninitialized urandom read (16 bytes read) [ 5.946285] systemd[1]: Initializing machine ID from random generator. [ 6.027317] random: ln: uninitialized urandom read (6 bytes read) [ 7.427482] random: systemd: uninitialized urandom read (16 bytes read) [ 7.430307] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 7.434821] random: systemd: uninitialized urandom read (16 bytes read) [ 7.437615] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 7.441957] random: systemd: uninitialized urandom read (16 bytes read) [ 7.445167] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.574167] device-mapper: uevent: version 1.0.3 [ 9.576525] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 10.734248] random: fast init done [ 10.848824] virtio_net virtio0 ens2: renamed from eth0 [ 17.793052] random: crng init done [ 17.800320] random: 5 urandom warning(s) missed due to ratelimiting [ 17.849884] scsi host0: ata_piix [ 17.878580] scsi host1: ata_piix [ 17.879908] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 17.884087] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 21.258053] dracut-initqueue[589]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 23.068560] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 26.559153] printk: systemd: 26 output lines suppressed due to ratelimiting [ 26.992056] SELinux: Disabled at runtime. [ 27.054892] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 27.067290] systemd[1]: Detected virtualization kvm. [ 27.069213] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 27.680183] systemd[1]: initrd-switch-root.service: Succeeded. [ 27.684085] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 27.696502] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 27.700910] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 27.704640] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 27.713525] systemd[1]: Starting Journal Service... Starting Journal Service... [ 27.720396] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-re[ 28.314601] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS mount-fs.service' for details. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 29.781877] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 31.670720] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 31.756778] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 32.766321] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 32.906475] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit)[ 40.200279] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit)[ 41.708483] NFS: Registering the id_resolver key type [ 41.733905] Key type id_resolver registered [ 41.735169] Key type id_legacy registered [*** ] A start job is running for Configur…only root support (14s / no limit) [** ] A start job is running for Configur…only root support (14s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg301-server login: [ 63.686293] hrtimer: interrupt took 3284204 ns [ 139.345413] libcfs: loading out-of-tree module taints kernel. [ 139.517681] Key type ._llcrypt registered [ 139.519304] Key type .llcrypt registered [ 139.804762] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 166.212480] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 168.676619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 168.688630] alg: No test for adler32 (adler32-zlib) [ 170.505021] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 171.823204] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 173.647156] Key type lgssc registered [ 175.926077] Lustre: Echo OBD driver; http://www.lustre.org/ [ 213.250188] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 273.649897] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 290.922546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 290.986909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 292.312511] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 292.346024] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 292.448655] Lustre: lustre-MDT0000: new disk, initializing [ 292.572990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 292.600923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 298.146692] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 324.552453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 324.775449] Lustre: 6546:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 324.868892] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 324.881230] Lustre: Skipped 1 previous similar message [ 325.021560] Lustre: lustre-MDT0001: new disk, initializing [ 325.173120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 325.277287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 325.336716] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 333.818941] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 339.967933] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 352.171461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 352.844237] Lustre: lustre-OST0000: new disk, initializing [ 352.861228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 352.885647] Lustre: 8485:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 352.998738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 362.522540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 362.535660] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 362.668616] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 363.819413] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 383.814141] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 384.009384] Lustre: lustre-OST0001: new disk, initializing [ 384.017911] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 384.034767] Lustre: 9559:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 384.163443] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 389.693381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 389.714218] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 389.822065] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 391.548643] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 411.137805] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 422.516202] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 434.064738] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing check_logdir /tmp/testlogs/ [ 442.372346] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing yml_node [ 449.323408] Lustre: DEBUG MARKER: Client: 2.17.56.2 [ 453.757340] Lustre: DEBUG MARKER: MDS: 2.17.56.2 [ 457.939532] Lustre: DEBUG MARKER: OSS: 2.17.56.2 [ 461.311459] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jul 30 01:18:50 EDT 2026 [ 489.133828] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 491.390499] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 512.506088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 512.509291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 512.536372] Lustre: Skipped 2 previous similar messages [ 512.569546] Lustre: Skipped 3 previous similar messages [ 516.930604] Lustre: server umount lustre-MDT0000 complete [ 522.678918] LustreError: 6538:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785388795 with bad export cookie 3407606847316888003 [ 522.689852] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 522.694694] LustreError: 6538:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 522.725592] LustreError: 10675:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 522.728651] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 522.737480] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 522.746392] LustreError: 10675:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 522.762760] Lustre: Skipped 2 previous similar messages [ 527.843376] LustreError: 10675:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 527.844842] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 527.881745] Lustre: Skipped 2 previous similar messages [ 529.856222] Lustre: server umount lustre-MDT0001 complete [ 535.435053] Lustre: server umount lustre-OST0000 complete [ 541.553777] Lustre: server umount lustre-OST0001 complete [ 550.468628] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 564.410548] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 613.389118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 613.930478] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 614.045792] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 614.187583] Lustre: lustre-MDT0000: new disk, initializing [ 614.319578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 614.341731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 619.669906] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 635.561323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 635.792336] Lustre: 15137:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 635.909798] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 635.914326] Lustre: Skipped 1 previous similar message [ 636.010320] Lustre: lustre-MDT0001: new disk, initializing [ 636.084310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 636.132086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 636.178473] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 642.917448] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 656.352496] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 666.240550] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 675.382071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 675.990358] Lustre: lustre-OST0000: new disk, initializing [ 676.020379] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 676.039692] Lustre: 17111:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 676.209764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 677.978962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 677.988815] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 678.177804] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 689.649436] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 702.053327] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 708.835313] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 709.186344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 718.492601] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 718.867520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 724.456753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 724.462383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 724.504339] Lustre: Skipped 1 previous similar message [ 724.531850] Lustre: Skipped 1 previous similar message [ 728.432704] Lustre: server umount lustre-OST0000 complete [ 734.690361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 734.696980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 734.721303] Lustre: Skipped 1 previous similar message [ 734.751700] Lustre: Skipped 1 previous similar message [ 741.513440] Lustre: server umount lustre-MDT0000 complete [ 744.938562] LustreError: 15149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 744.952451] LustreError: 15149:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 746.578523] LustreError: 15129:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785389019 with bad export cookie 3407606847316889536 [ 746.592837] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 746.593506] LustreError: 15129:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 747.361014] Lustre: server umount lustre-MDT0001 complete [ 753.164471] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:23:43 (1785389023) [ 756.546520] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 758.658059] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:23:49 (1785389029) [ 767.563996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 768.325640] LustreError: 19524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 768.422953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 773.616159] LustreError: 19525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 773.736269] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 783.843841] LustreError: 19524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 783.866645] LustreError: 19524:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 784.116440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 791.273804] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 803.897460] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 812.086220] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 822.100516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 832.558061] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 841.587946] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 854.506158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 854.507513] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 854.515683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 854.515692] Lustre: Skipped 4 previous similar messages [ 854.529145] Lustre: Skipped 1 previous similar message [ 858.553122] Lustre: server umount lustre-OST0000 complete [ 864.736900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 864.765715] Lustre: Skipped 1 previous similar message [ 870.193394] Lustre: server umount lustre-MDT0000 complete [ 874.959461] LustreError: 19506:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785389147 with bad export cookie 3407606847316890411 [ 874.967640] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 874.974154] LustreError: 19506:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 875.591847] Lustre: server umount lustre-MDT0001 complete [ 891.804320] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 896.414753] Key type lgssc unregistered [ 896.917807] LNet: 23061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.934249] LNetError: 23061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 896.954796] LNet: Removed LNI 192.168.203.101@tcp [ 898.934213] Key type .llcrypt unregistered [ 898.936613] Key type ._llcrypt unregistered [ 933.823205] Key type ._llcrypt registered [ 933.826412] Key type .llcrypt registered [ 934.034990] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 935.020474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 935.179318] alg: No test for adler32 (adler32-zlib) [ 936.684976] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 937.066949] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 938.743148] Key type lgssc registered [ 940.719645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 952.020450] loop: module loaded [ 952.133071] loop0: detected capacity change from 0 to 204800000 [ 952.413465] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 952.424898] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 952.819624] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 960.853817] loop0: detected capacity change from 0 to 204800000 [ 960.966422] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 960.985403] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 960.995995] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 968.634822] loop0: detected capacity change from 0 to 204800000 [ 976.189814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 976.240599] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 977.667818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 977.696606] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 977.748231] Lustre: lustre-MDT0000: new disk, initializing [ 977.800375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 977.818879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 984.807953] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 997.843249] loop1: detected capacity change from 0 to 204800000 [ 1007.793338] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1008.561820] Lustre: lustre-OST07c6: new disk, initializing [ 1008.573734] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 1008.587531] Lustre: Skipped 1 previous similar message [ 1008.591604] Lustre: 26265:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1008.789627] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 1016.999293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 1017.050291] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 1017.102866] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 1021.913163] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1030.616494] Lustre: 25252:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 1030.627242] Lustre: 25252:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 1030.635958] Lustre: 25252:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 1030.644710] Lustre: 25252:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1030.657512] Lustre: 25252:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 1030.670565] Lustre: 25252:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1047.969735] Lustre: 25251:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 1047.996655] Lustre: 25251:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 1048.017221] Lustre: 25251:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1048.045771] Lustre: 25251:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 1048.080415] Lustre: 25251:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 1048.099611] Lustre: 25251:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1055.431225] Lustre: 25252:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 1055.463876] Lustre: 25252:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 1055.484588] Lustre: 25252:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 1055.510691] Lustre: 25252:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1055.538865] Lustre: 25252:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1055.568681] Lustre: 25252:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1055.598433] Lustre: 25252:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 1055.617810] Lustre: 25252:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1055.641695] Lustre: 25252:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 1055.673860] Lustre: 25252:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1055.702291] Lustre: 25252:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1055.728103] Lustre: 25252:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1059.030692] Lustre: 25252:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 1059.044895] Lustre: 25252:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 1059.059842] Lustre: 25252:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1059.075088] Lustre: 25252:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1059.087391] Lustre: 25252:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1059.099398] Lustre: 25252:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1059.113942] Lustre: 25252:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 1059.126864] Lustre: 25252:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1059.136915] Lustre: 25252:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 1059.147154] Lustre: 25252:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1059.157315] Lustre: 25252:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1059.167049] Lustre: 25252:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1067.816398] Lustre: 26475:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 1067.835750] Lustre: 26475:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 1067.845017] Lustre: 26475:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 1067.856726] Lustre: 26475:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1067.872289] Lustre: 26475:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1067.881724] Lustre: 26475:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1067.890927] Lustre: 26475:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 1067.901818] Lustre: 26475:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1067.920023] Lustre: 26475:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 1067.932693] Lustre: 26475:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1067.944390] Lustre: 26475:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1067.962816] Lustre: 26475:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 1080.529314] Lustre: 26475:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 1080.554575] Lustre: 26475:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 1080.564131] Lustre: 26475:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 1080.595057] Lustre: 26475:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1080.611059] Lustre: 26475:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1080.628659] Lustre: 26475:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1080.643163] Lustre: 26475:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 1080.654808] Lustre: 26475:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1080.670628] Lustre: 26475:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 1080.687798] Lustre: 26475:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1080.699061] Lustre: 26475:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1080.708831] Lustre: 26475:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1134.565661] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1134.591218] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 1139.283283] Lustre: server umount lustre-OST07c6 complete [ 1154.413594] Lustre: server umount lustre-MDT0000 complete [ 1179.456601] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1182.782521] Key type lgssc unregistered [ 1183.136692] LNet: 28847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1183.148404] LNetError: 28847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1183.176773] LNet: Removed LNI 192.168.203.101@tcp [ 1184.330899] Key type .llcrypt unregistered [ 1184.334736] Key type ._llcrypt unregistered [ 1211.513976] Key type ._llcrypt registered [ 1211.519572] Key type .llcrypt registered [ 1211.737285] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1213.990896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1214.559867] alg: No test for adler32 (adler32-zlib) [ 1215.785547] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 1215.982555] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 1217.674044] Key type lgssc registered [ 1219.552173] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1230.613511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1230.652894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1232.445433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1239.332186] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1242.594104] LustreError: 30120:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1242.617973] LustreError: 30120:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1247.716487] LustreError: 30121:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1249.924481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1250.419251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1256.768532] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1266.686735] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1273.355372] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1281.934881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1282.270048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1291.166931] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1302.323306] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1302.774555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1320.416463] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1320.419713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1320.442796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1323.491094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1323.519945] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1323.534533] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1324.885758] Lustre: server umount lustre-OST0000 complete [ 1332.192558] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1332.209304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1332.232326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1332.234928] Lustre: Skipped 1 previous similar message [ 1336.462158] Lustre: server umount lustre-MDT0000 complete [ 1338.856849] LustreError: 30850:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1338.905144] LustreError: 30850:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1342.510066] LustreError: 30100:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785389615 with bad export cookie 8944783093241238431 [ 1342.515452] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1342.524228] LustreError: 30100:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1343.207281] Lustre: server umount lustre-MDT0001 complete [ 1362.797696] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1366.349511] Key type lgssc unregistered [ 1366.708722] LNet: 33880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1366.730600] LNetError: 33880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1366.756744] LNet: Removed LNI 192.168.203.101@tcp [ 1367.884637] Key type .llcrypt unregistered [ 1367.886312] Key type ._llcrypt unregistered [ 1393.017571] Key type ._llcrypt registered [ 1393.020446] Key type .llcrypt registered [ 1393.200891] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1396.249063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1396.335190] alg: No test for adler32 (adler32-zlib) [ 1397.759904] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 1398.191816] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 1399.967162] Key type lgssc registered [ 1402.194797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1413.911373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1413.949962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1415.756294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1422.053281] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1425.890642] LustreError: 35152:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1425.910306] LustreError: 35152:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1431.030330] LustreError: 35153:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1436.128105] LustreError: 35152:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1436.505954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1436.781393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1444.553154] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1456.463218] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1466.716981] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1475.825525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1476.446308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1485.695672] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1487.094673] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1497.272151] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1516.514958] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1516.532776] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1516.540072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1516.582187] Lustre: Skipped 1 previous similar message [ 1519.132810] Lustre: server umount lustre-OST0000 complete [ 1526.753075] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1526.757681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1526.765462] Lustre: Skipped 1 previous similar message [ 1526.782624] Lustre: Skipped 2 previous similar messages [ 1531.919327] LustreError: 35148:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1531.949236] LustreError: 35148:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1532.103550] Lustre: server umount lustre-MDT0000 complete [ 1536.996278] LustreError: 36628:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1537.027225] LustreError: 36628:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1537.151077] LustreError: 35133:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785389809 with bad export cookie 10264011100939889902 [ 1537.158208] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1537.169994] LustreError: 35133:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1538.342490] Lustre: server umount lustre-MDT0001 complete [ 1552.776851] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1556.455729] Key type lgssc unregistered [ 1557.092956] LNet: 38711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1557.110308] LNetError: 38711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1557.163513] LNet: Removed LNI 192.168.203.101@tcp [ 1558.715834] Key type .llcrypt unregistered [ 1558.719835] Key type ._llcrypt unregistered [ 1590.549786] Key type ._llcrypt registered [ 1590.551798] Key type .llcrypt registered [ 1590.656777] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1591.705464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1591.790807] alg: No test for adler32 (adler32-zlib) [ 1592.941763] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 1593.340540] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 1595.071138] Key type lgssc registered [ 1597.285784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1607.541831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1607.569324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1609.015624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1613.796792] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1618.400802] LustreError: 40129:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1618.419380] LustreError: 40129:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1623.531617] LustreError: 40130:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1625.227687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1626.000213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1631.639348] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1641.510704] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1651.799484] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1661.457364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1662.033241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1671.109268] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1678.396446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1680.433665] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1688.545164] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1688.554197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1688.567861] Lustre: Skipped 1 previous similar message [ 1692.422812] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.1@tcp (stopping) [ 1692.438137] Lustre: Skipped 2 previous similar messages [ 1693.668385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1693.689702] Lustre: Skipped 2 previous similar messages [ 1695.113053] Lustre: server umount lustre-MDT0000 complete [ 1697.490679] LustreError: 41401:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1700.843047] LustreError: 40111:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785389973 with bad export cookie 12674027825548734699 [ 1700.853055] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1700.883647] LustreError: 40111:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1702.606261] LustreError: 40124:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1702.627140] LustreError: 40124:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1702.629793] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.1@tcp (stopping) [ 1703.910298] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1703.940759] Lustre: Skipped 1 previous similar message [ 1707.724744] LustreError: 42548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1707.759610] LustreError: 42548:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1707.984959] Lustre: server umount lustre-MDT0001 complete [ 1727.114584] Lustre: server umount lustre-OST0000 complete [ 1761.011275] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1765.306440] Key type lgssc unregistered [ 1765.792597] LNet: 44019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1765.811708] LNetError: 44019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1765.851809] LNet: Removed LNI 192.168.203.101@tcp [ 1768.080558] Key type .llcrypt unregistered [ 1768.082403] Key type ._llcrypt unregistered [ 1806.821943] Key type ._llcrypt registered [ 1806.826790] Key type .llcrypt registered [ 1807.048089] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1808.982359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1809.081933] alg: No test for adler32 (adler32-zlib) [ 1810.241657] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 1810.509276] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 1812.207992] Key type lgssc registered [ 1813.342631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1824.185285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1824.242149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1826.269371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1834.303560] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1835.497147] LustreError: 45438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1835.545905] LustreError: 45438:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1840.629686] LustreError: 45437:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1845.738418] LustreError: 45438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1847.818146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1848.435646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1855.062684] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1866.251983] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1874.444611] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1882.902588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1883.316685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1891.916066] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1895.679503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1902.929776] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1911.266733] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1911.276097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1911.287053] Lustre: Skipped 1 previous similar message [ 1911.318583] Lustre: Skipped 1 previous similar message [ 1915.128164] Lustre: lustre-OST0000: Not available for connect from 192.168.203.1@tcp (stopping) [ 1916.391988] LustreError: 47259:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1916.418629] LustreError: 47259:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1916.594636] Lustre: server umount lustre-OST0000 complete [ 1937.400440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1937.414764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1937.433776] Lustre: Skipped 1 previous similar message [ 1937.462834] Lustre: Skipped 1 previous similar message [ 1941.536840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1944.093523] Lustre: server umount lustre-MDT0000 complete [ 1946.604383] LustreError: 46524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1946.640729] LustreError: 46524:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1949.851946] LustreError: 45420:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785390222 with bad export cookie 2997938093365217088 [ 1949.861509] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1949.872856] LustreError: 45420:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1956.683658] Lustre: server umount lustre-MDT0001 complete [ 1970.478098] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1975.041115] Key type lgssc unregistered [ 1975.529031] LNet: 49149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1975.534869] LNetError: 49149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1975.559182] LNet: Removed LNI 192.168.203.101@tcp [ 1976.912075] Key type .llcrypt unregistered [ 1976.918292] Key type ._llcrypt unregistered [ 2005.893616] Key type ._llcrypt registered [ 2005.896790] Key type .llcrypt registered [ 2006.010221] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 2008.095015] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2008.220886] alg: No test for adler32 (adler32-zlib) [ 2009.487709] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 2009.915768] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 2011.623449] Key type lgssc registered [ 2013.277292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2023.349994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2023.416730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2025.148542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2031.806943] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2034.658938] LustreError: 50566:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2034.690791] LustreError: 50566:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2039.776644] LustreError: 50567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2044.642503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2044.896643] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2045.235364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2052.060866] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2061.431534] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2068.784897] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2078.207062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2078.868668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2088.446108] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2096.323674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2098.676224] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2107.045310] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2112.283421] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 2125.750906] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-30 1h46m34s [ 2128.281707] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 2131.725782] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.1@tcp (stopping) [ 2132.448869] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2132.464373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2132.496152] Lustre: Skipped 2 previous similar messages [ 2132.806578] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.1@tcp (stopping) [ 2132.831656] Lustre: Skipped 3 previous similar messages [ 2138.058949] Lustre: server umount lustre-MDT0000 complete [ 2139.336520] LustreError: 51488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2143.149280] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 2145.510577] LustreError: 50561:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2145.556257] LustreError: 50561:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2151.628883] LustreError: 50562:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2151.656512] LustreError: 50562:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2152.704296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2152.900905] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2153.281906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2153.412328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2154.300526] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2159.289741] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2163.919608] Lustre: DEBUG MARKER: Wait for df (36783) ... [ 2165.962915] Lustre: DEBUG MARKER: done [ 2173.940828] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2173.960243] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2173.991244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2174.002787] Lustre: Skipped 3 previous similar messages [ 2174.962897] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2180.067699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2180.088506] Lustre: Skipped 3 previous similar messages [ 2187.231950] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2187.628219] Lustre: server umount lustre-OST0000 complete [ 2195.428599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2195.431351] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2195.435701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2195.457987] Lustre: Skipped 1 previous similar message [ 2195.500920] Lustre: Skipped 3 previous similar messages [ 2199.761109] Lustre: server umount lustre-MDT0000 complete [ 2203.131870] LustreError: 50566:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2203.164417] LustreError: 50566:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2204.860244] LustreError: 53197:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785390477 with bad export cookie 18031773557346397642 [ 2204.870071] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2204.877409] LustreError: 53197:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2205.463288] Lustre: server umount lustre-MDT0001 complete [ 2219.486160] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 2224.001749] Key type lgssc unregistered [ 2224.384690] LNet: 55500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2224.392418] LNetError: 55500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2224.416440] LNet: Removed LNI 192.168.203.101@tcp [ 2225.527283] Key type .llcrypt unregistered [ 2225.532703] Key type ._llcrypt unregistered [ 2280.835500] Key type ._llcrypt registered [ 2280.837307] Key type .llcrypt registered [ 2280.965274] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 2282.466634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2282.615668] alg: No test for adler32 (adler32-zlib) [ 2284.209262] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 2284.408534] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 2286.047153] Key type lgssc registered [ 2288.327293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2299.047311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2299.073670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2300.330493] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2300.370042] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2300.607454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2309.136857] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2322.463962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2322.760737] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2322.853500] Lustre: 57718:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2323.597917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2332.202955] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2342.577682] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2351.351599] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2361.497670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2361.663863] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2361.907103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2370.707092] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2371.925248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2382.301209] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2391.529037] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-30 1h51m01s [ 2393.593401] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2403.432267] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2406.404478] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2406.472208] LustreError: 57725:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2426.635559] Lustre: lustre-MDT0000: Client 441ee519-051e-4bc1-bd12-f0fb959b1b38 (at 192.168.203.1@tcp) reconnecting [ 2446.543095] LustreError: 57725:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2450.170178] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2453.460604] Lustre: DEBUG MARKER: done [ 2464.228688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2464.231262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2464.236295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2464.264735] Lustre: Skipped 1 previous similar message [ 2464.290190] Lustre: Skipped 1 previous similar message [ 2468.900962] Lustre: server umount lustre-OST0000 complete [ 2474.466424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2474.481055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2474.496258] Lustre: Skipped 1 previous similar message [ 2474.521036] Lustre: Skipped 1 previous similar message [ 2475.552724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2475.559311] Lustre: Skipped 1 previous similar message [ 2480.199284] Lustre: server umount lustre-MDT0000 complete [ 2485.433995] LustreError: 57709:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785390758 with bad export cookie 12358573211710442490 [ 2485.444342] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2485.454928] LustreError: 57709:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2486.106846] Lustre: server umount lustre-MDT0001 complete [ 2504.770338] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 2509.420956] Key type lgssc unregistered [ 2509.781716] LNet: 62068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2509.791471] LNetError: 62068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2509.806082] LNet: Removed LNI 192.168.203.101@tcp [ 2511.079226] Key type .llcrypt unregistered [ 2511.084580] Key type ._llcrypt unregistered [ 2571.288591] Key type ._llcrypt registered [ 2571.291953] Key type .llcrypt registered [ 2571.452613] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 2573.252129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2573.296125] alg: No test for adler32 (adler32-zlib) [ 2574.280991] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 2574.513138] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 2576.280249] Key type lgssc registered [ 2578.316765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2582.707455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2582.732259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2584.157649] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2584.223755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2584.526602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2585.672530] Lustre: Failing over lustre-MDT0000 [ 2586.321950] Lustre: server umount lustre-MDT0000 complete [ 2594.049823] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:54:25 (1785390865) [ 2603.961793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2604.678731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2611.544825] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2623.372195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2623.576427] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2623.634807] Lustre: 64868:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2624.148199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2631.750958] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2643.245409] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2652.548075] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2663.890477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2664.354590] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2664.879598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2670.392914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2676.251546] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2693.890619] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2701.487956] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2714.557557] Lustre: server umount lustre-MDT0000 complete [ 2716.142628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2716.161069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2720.205023] LustreError: 65606:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785390992 with bad export cookie 4827340487576465598 [ 2720.219480] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2720.228624] LustreError: 65606:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2721.159817] Lustre: server umount lustre-MDT0001 complete [ 2725.437179] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2732.971694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2738.143303] Lustre: 63793:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785390994/real 1785390994] req@0000000010b0c22d x1872117992832256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785391010 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2738.187331] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2738.207335] Lustre: Skipped 2 previous similar messages [ 2742.241366] Lustre: 63794:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785390998/real 1785390998] req@0000000067665310 x1872117992832384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785391014 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2743.264513] Lustre: 63792:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785390999/real 1785390999] req@00000000ba4d2543 x1872117992832512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785391015 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2745.134249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2745.318172] Lustre: 63791:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785391002/real 1785391002] req@00000000b51d5dc5 x1872117992832768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785391018 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2750.496167] Lustre: 63794:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785391007/real 1785391007] req@000000006af3f13d x1872117992832896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785391023 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2764.834184] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x42fe2898dbdb39a1 [ 2764.861589] Lustre: MGC192.168.203.101@tcp: Connection restored to 0@lo (at 0@lo) [ 2765.418119] LustreError: 66721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2765.451812] LustreError: 66721:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2765.552683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2765.699667] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2767.588089] LustreError: 66722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2771.377235] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2772.966364] LustreError: 66721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2778.081234] LustreError: 66722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2782.192415] LustreError: 66721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2782.206393] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2783.797843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2784.419569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2789.881731] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2790.060940] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2800.721974] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2808.701726] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2820.064632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2820.085580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2820.119829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2825.697513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2825.714550] Lustre: Skipped 4 previous similar messages [ 2830.819208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2830.836876] Lustre: Skipped 2 previous similar messages [ 2831.327307] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2831.956398] Lustre: server umount lustre-MDT0000 complete [ 2835.936384] LustreError: 68651:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2835.957396] LustreError: 68651:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2838.935684] LustreError: 64861:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785391111 with bad export cookie 4827340487576467873 [ 2838.938974] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2838.955269] LustreError: 64861:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2840.417820] Lustre: server umount lustre-MDT0001 complete [ 2850.448027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2856.416276] Lustre: 63794:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785391113/real 1785391113] req@000000009a013090 x1872117992863360/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785391129 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2856.451981] Lustre: 63794:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2856.462902] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2856.485699] Lustre: Skipped 2 previous similar messages [ 2862.546547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2863.648626] LustreError: 63790:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001aea25e6 x1872117992864256/t0(0) o250->MGC192.168.203.101@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2863.991290] LustreError: 66722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2864.109365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2864.214712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2866.184635] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2870.442108] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2883.278453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2889.203802] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2889.795658] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2898.993413] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2906.579594] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2919.906576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2919.912248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2919.920093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2919.953907] Lustre: Skipped 2 previous similar messages [ 2919.956281] Lustre: Skipped 1 previous similar message [ 2923.050268] Lustre: server umount lustre-MDT0000 complete [ 2925.025626] LustreError: 72072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2925.054170] LustreError: 72072:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 2928.525174] LustreError: 64862:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785391201 with bad export cookie 4827340487576469252 [ 2928.532072] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2928.536229] LustreError: 64862:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2928.929851] Lustre: server umount lustre-MDT0001 complete [ 2937.861883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2942.854561] Lustre: DEBUG MARKER: files compared the same [ 2945.503091] Lustre: 63792:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785391202/real 1785391202] req@00000000926e01e1 x1872117992893952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785391218 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2945.521164] Lustre: 63792:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 2945.526490] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2946.274776] Lustre: server umount lustre-OST0000 complete [ 2971.062270] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 2975.579779] Key type lgssc unregistered [ 2976.110639] LNet: 74615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2976.122380] LNetError: 74615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2977.189344] LNet: Removed LNI 192.168.203.101@tcp [ 2979.116335] Key type .llcrypt unregistered [ 2979.122217] Key type ._llcrypt unregistered [ 3017.570169] Key type ._llcrypt registered [ 3017.575952] Key type .llcrypt registered [ 3017.730161] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 3019.551659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3019.796287] alg: No test for adler32 (adler32-zlib) [ 3021.069672] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 3021.715891] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 3023.447148] Key type lgssc registered [ 3024.594588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3035.194709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3035.234238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3036.536607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3042.685137] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug +malloc all [ 3046.883268] LustreError: 76048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3046.910158] LustreError: 76048:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3052.023773] LustreError: 76049:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3055.111329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3055.808554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3062.996302] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug +malloc all [ 3074.452323] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3085.244361] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3097.951985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3098.525033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3106.762727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 3110.833843] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug +malloc all [ 3122.394286] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3137.507595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3137.517896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3137.537869] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3137.543879] Lustre: Skipped 1 previous similar message [ 3137.614669] Lustre: Skipped 1 previous similar message [ 3139.410548] Lustre: server umount lustre-OST0000 complete [ 3147.744068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3147.751063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3147.756278] Lustre: Skipped 1 previous similar message [ 3147.772432] Lustre: Skipped 1 previous similar message [ 3150.863333] Lustre: server umount lustre-MDT0000 complete [ 3152.864719] LustreError: 76049:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3152.874499] LustreError: 76049:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3155.268503] LustreError: 76030:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785391427 with bad export cookie 2969288515058357249 [ 3155.276666] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3155.284264] LustreError: 76030:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3156.074470] Lustre: server umount lustre-MDT0001 complete [ 3169.936741] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 3173.675352] Key type lgssc unregistered [ 3173.992739] LNet: 79588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3174.009151] LNetError: 79588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3174.046628] LNet: Removed LNI 192.168.203.101@tcp [ 3175.180242] Key type .llcrypt unregistered [ 3175.182618] Key type ._llcrypt unregistered [ 3204.816474] Key type ._llcrypt registered [ 3204.820529] Key type .llcrypt registered [ 3204.975858] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 3206.912950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3207.079203] alg: No test for adler32 (adler32-zlib) [ 3208.478870] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 3208.842414] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 3210.784400] Key type lgssc registered [ 3212.557261] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3224.908257] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3224.956268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3240.735481] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3245.855289] LustreError: 80982:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.101@tcp: failed processing log, type 4: rc = -110 [ 3272.479158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3281.151782] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3293.120472] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3302.894913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3303.090252] Lustre: *** cfs_fail_loc=706, val=0*** [ 3303.846611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3303.956462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 3310.210486] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3312.102319] LustreError: 82070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3312.123930] LustreError: 82070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3317.223307] LustreError: 82068:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3317.244829] LustreError: 82068:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3321.185192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3321.832718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3327.104479] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3336.727971] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3343.574611] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3347.431615] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3347.437627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3347.492346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3352.571785] LustreError: 80996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3352.984317] Lustre: server umount lustre-OST0000 complete [ 3362.796107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3362.807079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3362.826434] Lustre: Skipped 2 previous similar messages [ 3362.835088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3362.847540] Lustre: Skipped 2 previous similar messages [ 3364.860430] Lustre: server umount lustre-MDT0000 complete [ 3367.906019] LustreError: 82073:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3367.933341] LustreError: 82073:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3369.925304] LustreError: 80990:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785391642 with bad export cookie 16482426550133475528 [ 3369.930342] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3369.941257] LustreError: 80990:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3370.529796] Lustre: server umount lustre-MDT0001 complete [ 3385.306264] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 3390.352865] Key type lgssc unregistered [ 3390.839469] LNet: 84601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3390.847883] LNetError: 84601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3390.882726] LNet: Removed LNI 192.168.203.101@tcp [ 3392.159337] Key type .llcrypt unregistered [ 3392.163357] Key type ._llcrypt unregistered [ 3431.710848] Key type ._llcrypt registered [ 3431.715948] Key type .llcrypt registered [ 3431.902339] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 3433.034873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3433.127220] alg: No test for adler32 (adler32-zlib) [ 3434.227588] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 3434.610904] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 3436.319746] Key type lgssc registered [ 3437.493846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3448.179463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3448.207571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3455.121832] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3464.795925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3465.618307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3470.818773] LustreError: 86709:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3470.842092] LustreError: 86709:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3472.012589] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3475.944278] LustreError: 86713:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3475.969312] LustreError: 86713:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3481.055930] LustreError: 86712:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3481.078125] LustreError: 86712:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3481.611306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3482.124026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3487.214365] LustreError: 86713:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3487.235705] LustreError: 86713:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3489.476226] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3495.393321] LustreError: 86713:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3495.432926] LustreError: 86713:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3499.734456] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3499.763437] Lustre: MGS: Client d24b093c-ecc5-4a27-944c-6e2c4d3aa3ff (at 0@lo) reconnecting [ 3499.821338] LustreError: 86723:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3500.219339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3500.324724] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 3506.010571] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3526.118961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3526.125997] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3526.130954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3526.155303] Lustre: Skipped 1 previous similar message [ 3529.363573] Lustre: server umount lustre-OST0000 complete [ 3536.356180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3536.362221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3536.399613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3536.410079] Lustre: Skipped 2 previous similar messages [ 3540.381663] Lustre: server umount lustre-MDT0000 complete [ 3541.473188] LustreError: 86709:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3541.495052] LustreError: 86709:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3545.342563] Lustre: server umount lustre-MDT0001 complete [ 3550.895445] Lustre: server umount MGS complete [ 3573.895968] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 02:10:43 (1785391843) [ 3620.514332] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 3641.692824] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 3696.497133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3703.860793] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3718.912080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3719.162583] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3719.204062] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3719.304882] Lustre: lustre-MDT0001: new disk, initializing [ 3719.424981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3719.475867] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3719.493629] Lustre: 93931:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3719.504799] LustreError: 93931:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3726.162886] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3740.277922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3740.684079] Lustre: lustre-OST0000: new disk, initializing [ 3740.688945] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3740.698313] Lustre: 94697:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3740.839139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3749.476563] Lustre: 94987:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000079ae37de x1872118894602752/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1785392038 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3751.027663] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3763.095368] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3769.823592] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3772.671755] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3772.689780] Lustre: MGS: Client 2891b06b-b23a-490f-8b05-13f81360af0d (at 0@lo) reconnecting [ 3772.764815] Lustre: 93196:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3772.800700] LustreError: 93917:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3772.843891] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3772.957094] Lustre: lustre-MDT0000: new disk, initializing [ 3773.125204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3773.174473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3774.240071] Lustre: 95769:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000f6d23e2c x1872118894610432/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1785392062 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3779.302030] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3779.466847] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3783.342696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3783.356332] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3783.446209] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3809.249845] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3809.253154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3809.279569] Lustre: Skipped 1 previous similar message [ 3809.285520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3812.840840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3812.850183] Lustre: Skipped 1 previous similar message [ 3815.016391] Lustre: server umount lustre-OST0000 complete [ 3823.081891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3823.091985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3823.110684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3823.110692] LustreError: Skipped 1 previous similar message [ 3823.122392] Lustre: Skipped 2 previous similar messages [ 3823.149963] Lustre: Skipped 1 previous similar message [ 3826.110314] Lustre: server umount lustre-MDT0000 complete [ 3828.207473] LustreError: 96194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3828.225427] LustreError: 96194:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3845.087151] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3845.603507] Lustre: server umount lustre-MDT0001 complete [ 3850.338314] Lustre: server umount MGS complete [ 3864.716336] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 02:15:35 (1785392135) [ 3890.498118] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 3894.017178] Key type lgssc unregistered [ 3894.480672] LNet: 98269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3894.501080] LNetError: 98269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3894.543169] LNet: Removed LNI 192.168.203.101@tcp [ 3895.980632] Key type .llcrypt unregistered [ 3895.983054] Key type ._llcrypt unregistered [ 3916.713533] Key type ._llcrypt registered [ 3916.716233] Key type .llcrypt registered [ 3916.841175] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 3918.027574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3918.212203] alg: No test for adler32 (adler32-zlib) [ 3919.432177] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 3919.767124] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 3921.424191] Key type lgssc registered [ 3923.166244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3936.801577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3936.834897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3938.315500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3944.692869] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3947.491248] LustreError: 99496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3947.513683] LustreError: 99496:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3952.613929] LustreError: 99497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3957.753037] LustreError: 99496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3958.014641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3959.286803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3966.725368] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3978.946804] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3991.072461] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4004.240214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4004.942528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4008.057669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 4014.313525] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4024.256802] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4039.136585] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4039.147644] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4039.161218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4040.171805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4040.173405] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4040.189187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4040.196186] Lustre: Skipped 1 previous similar message [ 4045.161353] Lustre: server umount lustre-OST0000 complete [ 4050.400627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4050.401938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4050.416287] Lustre: Skipped 1 previous similar message [ 4050.438723] Lustre: Skipped 1 previous similar message [ 4055.502640] Lustre: server umount lustre-MDT0000 complete [ 4055.521369] LustreError: 99497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4055.541077] LustreError: 99497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4059.622530] LustreError: 100827:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4059.656506] LustreError: 100827:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4059.682188] LustreError: 99476:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785392332 with bad export cookie 15658702606329158481 [ 4059.685034] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4059.704669] LustreError: 99476:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4060.619326] Lustre: server umount lustre-MDT0001 complete [ 4075.324936] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 4079.504219] Key type lgssc unregistered [ 4079.851643] LNet: 103055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4079.871226] LNetError: 103055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4080.937969] LNet: Removed LNI 192.168.203.101@tcp [ 4082.356256] Key type .llcrypt unregistered [ 4082.360364] Key type ._llcrypt unregistered [ 4101.678608] Key type ._llcrypt registered [ 4101.680594] Key type .llcrypt registered [ 4101.795397] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 4103.000547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4103.194355] alg: No test for adler32 (adler32-zlib) [ 4104.354726] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 4104.618848] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 4106.335601] Key type lgssc registered [ 4108.119842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4117.094239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4117.116493] LustreError: 104135:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 4117.146169] LustreError: 104142:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 4117.166023] LustreError: 104135:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4972 [ 4117.180813] LustreError: 104135:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 4117.188025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4117.196466] LustreError: 104135:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 4119.121092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4129.253508] LustreError: 104180:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4129.293507] LustreError: 104180:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4129.408509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4129.949277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4136.488870] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4143.330946] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing lsmod [ 4146.874671] LustreError: 105337:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 4146.963938] LustreError: 105348:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 4146.976363] LustreError: 105337:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4910 [ 4146.988951] LustreError: 105337:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 4146.998436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4147.003200] LustreError: 105337:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 4147.714653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4152.810292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4152.813869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4152.834522] Lustre: Skipped 2 previous similar messages [ 4152.846517] Lustre: Skipped 1 previous similar message [ 4154.914475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4154.933681] Lustre: Skipped 3 previous similar messages [ 4159.971110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4165.092383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4165.114650] Lustre: Skipped 4 previous similar messages [ 4166.112213] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4166.647897] Lustre: server umount lustre-MDT0000 complete [ 4170.216532] LustreError: 104179:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4170.240675] LustreError: 104179:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4171.116023] LustreError: 104156:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785392443 with bad export cookie 17520113029543412215 [ 4171.131333] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4171.138373] LustreError: 104156:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4171.313784] LustreError: 104611:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000035c103ec x1872119597128704/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4171.844428] Lustre: server umount lustre-MDT0001 complete [ 4187.233504] Lustre: server umount lustre-OST0000 complete [ 4197.981067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4198.741812] LustreError: 106414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4198.892754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4204.012634] LustreError: 106415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4204.508031] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4209.128122] LustreError: 106414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4215.819170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4216.324069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4222.253841] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4231.730717] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4239.276530] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4248.547150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4249.120653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4258.262527] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4268.307295] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4268.693340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 4279.265700] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4279.271152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4279.291910] Lustre: Skipped 1 previous similar message [ 4279.314630] Lustre: Skipped 1 previous similar message [ 4282.404069] Lustre: server umount lustre-OST0000 complete [ 4289.505780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4289.509900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4289.533408] Lustre: Skipped 1 previous similar message [ 4289.545282] Lustre: Skipped 1 previous similar message [ 4295.455442] Lustre: server umount lustre-MDT0000 complete [ 4299.745069] LustreError: 106409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4299.775034] LustreError: 106409:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4301.594463] LustreError: 106395:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785392574 with bad export cookie 17520113029543412915 [ 4301.601921] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4301.611621] LustreError: 106395:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4302.265168] Lustre: server umount lustre-MDT0001 complete [ 4320.396552] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 4325.233483] Key type lgssc unregistered [ 4325.789548] LNet: 109962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4325.821716] LNetError: 109962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4326.885945] LNet: Removed LNI 192.168.203.101@tcp [ 4328.274807] Key type .llcrypt unregistered [ 4328.280919] Key type ._llcrypt unregistered [ 4364.649451] Key type ._llcrypt registered [ 4364.654383] Key type .llcrypt registered [ 4364.819323] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 4366.256391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4366.583189] alg: No test for adler32 (adler32-zlib) [ 4367.899389] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 4368.505302] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 4370.306663] Key type lgssc registered [ 4371.731527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4384.801979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4384.855833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4386.608731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4393.757949] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4397.026879] LustreError: 111385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4397.043101] LustreError: 111385:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4402.158884] LustreError: 111384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4407.281371] LustreError: 111385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4407.813297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4408.623979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4415.107245] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4426.429834] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4433.742980] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4443.674276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4444.237640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4451.875915] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4456.526158] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 4461.031533] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4469.645889] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 4477.824237] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 4487.136833] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4487.152473] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4487.166334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4487.183321] Lustre: Skipped 1 previous similar message [ 4487.193768] Lustre: Skipped 1 previous similar message [ 4491.576588] Lustre: server umount lustre-OST0000 complete [ 4500.960430] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4500.982530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4501.032315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4502.515899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4502.525833] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4506.658909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4506.667679] Lustre: Skipped 2 previous similar messages [ 4511.712884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4511.719508] Lustre: Skipped 1 previous similar message [ 4512.223124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4512.709816] Lustre: server umount lustre-MDT0000 complete [ 4516.834157] LustreError: 111385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4516.864496] LustreError: 111385:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4519.329128] LustreError: 111363:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785392792 with bad export cookie 13583321288496198934 [ 4519.330576] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4519.350438] LustreError: 111363:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4520.825754] Lustre: server umount lustre-MDT0001 complete [ 4540.287753] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 4544.884778] Key type lgssc unregistered [ 4545.392895] LNet: 115043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4545.408176] LNetError: 115043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4545.466453] LNet: Removed LNI 192.168.203.101@tcp [ 4546.825588] Key type .llcrypt unregistered [ 4546.828134] Key type ._llcrypt unregistered [ 4574.100784] Key type ._llcrypt registered [ 4574.115385] Key type .llcrypt registered [ 4574.345433] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 4576.806395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4576.847494] alg: No test for adler32 (adler32-zlib) [ 4578.289241] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 4578.520819] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 4580.175203] Key type lgssc registered [ 4582.177523] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4593.699274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4593.710494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4595.176217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4601.556386] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4604.398838] LustreError: 116269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4604.435651] LustreError: 116269:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4609.506964] LustreError: 116270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4613.810502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4614.542343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4622.149845] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4633.150828] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4641.649224] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4654.099265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4654.419584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4664.757612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 4665.689530] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4679.102037] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4695.524285] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4695.526807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4695.554291] Lustre: Skipped 1 previous similar message [ 4695.574956] Lustre: Skipped 1 previous similar message [ 4699.223751] Lustre: server umount lustre-OST0000 complete [ 4705.761058] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4705.797854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4705.808243] Lustre: Skipped 1 previous similar message [ 4710.731489] Lustre: server umount lustre-MDT0000 complete [ 4710.898368] LustreError: 116264:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4710.913959] LustreError: 116264:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4715.386821] LustreError: 117004:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785392988 with bad export cookie 6958765774402553761 [ 4715.395461] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4715.405577] LustreError: 117004:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4715.999705] Lustre: server umount lustre-MDT0001 complete [ 4732.272218] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 4736.995806] Key type lgssc unregistered [ 4737.324553] LNet: 119829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4737.333834] LNetError: 119829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4737.357168] LNet: Removed LNI 192.168.203.101@tcp [ 4739.050105] Key type .llcrypt unregistered [ 4739.055892] Key type ._llcrypt unregistered [ 4774.751765] Key type ._llcrypt registered [ 4774.757050] Key type .llcrypt registered [ 4774.909398] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 4776.987404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4777.020538] alg: No test for adler32 (adler32-zlib) [ 4778.195955] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 4778.457674] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 4780.234459] Key type lgssc registered [ 4782.287824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4794.746683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4794.809155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4796.810492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4804.223976] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4807.152974] LustreError: 121249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4807.182174] LustreError: 121249:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4808.167648] LustreError: 121250:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4813.299371] LustreError: 121249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4815.664787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4816.034704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4820.940699] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4831.000169] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4840.685434] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4851.091565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4851.782323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4859.985975] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4868.117859] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4868.220098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4874.985502] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4879.682687] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4879.685485] Lustre: Skipped 2 previous similar messages [ 4884.960231] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4884.971809] Lustre: Skipped 2 previous similar messages [ 4885.198901] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4885.251593] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4893.536733] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.1@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4893.547897] Lustre: Skipped 2 previous similar messages [ 4895.241138] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.1@tcp 192.168.0.[2,10]@tcp [ 4900.965355] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.1/24@tcp in log lustre-MDT0000 [ 4900.978290] Lustre: Skipped 2 previous similar messages [ 4903.118184] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.1/24@tcp [ 4903.120912] Lustre: Skipped 1 previous similar message [ 4912.610239] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4912.619079] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4912.652718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4914.660092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4914.667560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4914.678488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4914.711023] Lustre: Skipped 1 previous similar message [ 4918.575084] Lustre: server umount lustre-OST0000 complete [ 4923.369415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4923.390064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4923.408979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4930.016159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4930.027966] Lustre: Skipped 3 previous similar messages [ 4935.136202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4935.146395] Lustre: Skipped 1 previous similar message [ 4937.183118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4937.568517] Lustre: server umount lustre-MDT0000 complete [ 4940.267575] LustreError: 121249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4940.293831] LustreError: 121249:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4942.679648] LustreError: 121229:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785393215 with bad export cookie 14537052077310240585 [ 4942.685328] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4942.689490] LustreError: 121229:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4943.361253] Lustre: server umount lustre-MDT0001 complete [ 4958.317552] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 4962.322828] Key type lgssc unregistered [ 4962.722561] LNet: 125900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4962.729569] LNetError: 125900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4962.750780] LNet: Removed LNI 192.168.203.101@tcp [ 4963.821210] Key type .llcrypt unregistered [ 4963.826412] Key type ._llcrypt unregistered [ 4999.086334] Key type ._llcrypt registered [ 4999.092649] Key type .llcrypt registered [ 4999.233304] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 5000.466429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5000.802437] alg: No test for adler32 (adler32-zlib) [ 5002.044775] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 5002.373384] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 5004.129944] Key type lgssc registered [ 5005.842282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5015.379930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5015.430573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5017.073581] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 5017.096928] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.1/24@tcp [ 5017.134210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5023.821501] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5027.310282] LustreError: 127468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5027.325609] LustreError: 127468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5032.417445] LustreError: 127467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5035.364631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5035.917382] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 5035.971973] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.1/24@tcp [ 5036.107965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5044.414484] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5053.880369] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5062.393322] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5071.371806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5071.974538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5080.989931] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5087.362447] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 5090.488656] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5107.172093] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5107.185031] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5107.220304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5108.193133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5108.197040] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5108.211176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5108.236229] Lustre: Skipped 1 previous similar message [ 5110.060550] Lustre: server umount lustre-OST0000 complete [ 5117.923962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5117.939238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5117.959406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5121.228707] Lustre: server umount lustre-MDT0000 complete [ 5123.552518] LustreError: 127463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5123.584390] LustreError: 127463:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5126.254246] LustreError: 127448:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785393398 with bad export cookie 15277714749399747080 [ 5126.259952] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5126.278176] LustreError: 127448:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5126.751890] Lustre: server umount lustre-MDT0001 complete [ 5143.429026] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 5148.470955] Key type lgssc unregistered [ 5149.163682] LNet: 131076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5149.178631] LNetError: 131076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5150.260864] LNet: Removed LNI 192.168.203.101@tcp [ 5152.541580] Key type .llcrypt unregistered [ 5152.548079] Key type ._llcrypt unregistered [ 5201.180476] Key type ._llcrypt registered [ 5201.186885] Key type .llcrypt registered [ 5201.555271] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 5230.937543] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 5232.661643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5232.710792] alg: No test for adler32 (adler32-zlib) [ 5233.799497] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 5234.037763] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 5235.727591] Key type lgssc registered [ 5236.726779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5278.172674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5278.213722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5279.533918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5279.564550] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5279.635624] Lustre: lustre-MDT0000: new disk, initializing [ 5279.712346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5279.735595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5285.739855] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5299.291919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5299.508268] Lustre: 134481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5299.597303] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5299.604202] Lustre: Skipped 1 previous similar message [ 5299.734987] Lustre: lustre-MDT0001: new disk, initializing [ 5299.870948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5299.914188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5299.929386] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5305.844411] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5318.312970] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5325.213321] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5334.554037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5335.042950] Lustre: lustre-OST0000: new disk, initializing [ 5335.045092] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5335.060352] Lustre: 136454:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5335.168430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5340.825372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5340.849260] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5341.091680] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5343.900218] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5358.954697] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5364.683975] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5365.020666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5373.302436] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5373.912447] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5382.114039] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5382.116325] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5382.128106] LustreError: Skipped 1 previous similar message [ 5382.129149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5382.182665] Lustre: Skipped 1 previous similar message [ 5387.237624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5387.252659] Lustre: Skipped 2 previous similar messages [ 5391.328056] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5391.573805] Lustre: server umount lustre-OST0000 complete [ 5396.962204] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5396.974062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5396.993953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5402.594690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5402.609878] Lustre: Skipped 3 previous similar messages [ 5403.378185] Lustre: server umount lustre-MDT0000 complete [ 5408.774600] LustreError: 135278:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785393681 with bad export cookie 2211133740722687821 [ 5408.777996] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5408.786472] LustreError: 135278:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5409.331691] Lustre: server umount lustre-MDT0001 complete [ 5420.421127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5421.133140] LustreError: 138674:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5421.154954] LustreError: 138674:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5421.269455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5426.659040] LustreError: 138675:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5426.688150] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5430.760570] LustreError: 138674:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5435.879620] LustreError: 138675:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5438.105579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5438.558810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5443.106025] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5452.944515] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5461.670651] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5471.598254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5471.900811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5479.938893] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5489.376325] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5495.908627] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5496.236990] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5502.261374] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5502.562996] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5514.750658] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5515.079953] Lustre: lustre-OST0001: new disk, initializing [ 5515.096530] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5515.114148] Lustre: 142051:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5515.263158] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5524.646058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5524.668269] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5524.758967] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5525.982302] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5539.736835] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5540.078627] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5547.328340] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5547.519741] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5557.431561] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93ed10460000.ost_server_uuid 50 [ 5560.349646] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff93ed10460000.ost_server_uuid in IDLE state after 0 sec [ 5581.286433] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5581.291510] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5581.297980] LustreError: Skipped 1 previous similar message [ 5581.308236] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5581.308750] Lustre: Skipped 2 previous similar messages [ 5581.335848] Lustre: Skipped 1 previous similar message [ 5591.011382] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5591.126472] Lustre: server umount lustre-OST0001 complete [ 5591.539719] LustreError: 140498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5596.650473] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5596.665472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5596.668438] Lustre: Skipped 1 previous similar message [ 5596.683673] Lustre: Skipped 3 previous similar messages [ 5601.781505] LustreError: 141169:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5601.824862] LustreError: 141169:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5602.450316] Lustre: server umount lustre-OST0000 complete [ 5607.399943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5607.405373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5613.699443] Lustre: server umount lustre-MDT0000 complete [ 5618.520924] LustreError: 138655:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785393891 with bad export cookie 2211133740722688710 [ 5618.528498] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5618.536996] LustreError: 138655:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5619.299840] Lustre: server umount lustre-MDT0001 complete [ 5651.929204] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 5655.708832] Key type lgssc unregistered [ 5656.086656] LNet: 145101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5656.099657] LNetError: 145101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5656.132974] LNet: Removed LNI 192.168.203.101@tcp [ 5657.121406] Key type .llcrypt unregistered [ 5657.127062] Key type ._llcrypt unregistered [ 5719.423346] Key type ._llcrypt registered [ 5719.427467] Key type .llcrypt registered [ 5719.524565] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 5744.719736] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 5747.083987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5747.120710] alg: No test for adler32 (adler32-zlib) [ 5748.123155] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 5748.308873] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 5750.151277] Key type lgssc registered [ 5751.950783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5799.999089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5800.127855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5801.528122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5801.583467] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5801.694930] Lustre: lustre-MDT0000: new disk, initializing [ 5801.807326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5801.833715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5809.771924] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5825.599891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5825.737989] Lustre: 149265:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5825.798164] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5825.805605] Lustre: Skipped 1 previous similar message [ 5825.871376] Lustre: lustre-MDT0001: new disk, initializing [ 5825.986643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5826.021696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5826.046380] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5831.730516] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5843.848865] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5852.114339] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5862.730758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5863.022923] Lustre: lustre-OST0000: new disk, initializing [ 5863.026250] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5863.036278] Lustre: 151235:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5863.160887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5868.682492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5868.706768] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5868.979764] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5872.007038] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5885.064534] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5889.750447] Lustre: Failing over lustre-OST0000 [ 5890.313552] Lustre: server umount lustre-OST0000 complete [ 5907.999794] Lustre: 147484:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785394164/real 1785394164] req@00000000d5aa601e x1872121319953152/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1785394180 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5908.048231] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5911.007081] Lustre: 147486:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785394167/real 1785394167] req@00000000365ad893 x1872121319954560/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785394183 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5911.027833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5911.037255] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5911.573148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5911.600224] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 5912.058509] Lustre: 147485:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785394167/real 1785394167] req@000000004183eacd x1872121319954432/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785394183 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5912.098578] Lustre: 147485:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5915.616163] Lustre: 147483:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785394172/real 1785394172] req@00000000515f8c32 x1872121319955712/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785394188 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5915.676149] Lustre: 147483:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5917.761388] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5920.373569] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5920.871765] Lustre: 147483:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785394177/real 1785394177] req@0000000013d78dfb x1872121319956864/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785394193 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5920.956672] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 5920.959520] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5925.399487] Lustre: Failing over lustre-MDT0000 [ 5926.002234] Lustre: server umount lustre-MDT0000 complete [ 5928.434558] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5928.436180] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5928.494337] Lustre: Skipped 2 previous similar messages [ 5933.252835] LustreError: 149270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5933.295757] LustreError: 149270:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5938.392542] LustreError: 151669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5938.414315] LustreError: 151669:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5943.495657] LustreError: 149272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5943.531427] LustreError: 149272:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5944.673433] Lustre: 147486:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785394201/real 1785394201] req@00000000f4ffdf08 x1872121319970048/t0(0) o400->MGC192.168.203.101@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785394217 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5944.700135] Lustre: 147486:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5944.707794] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5947.394152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5948.615334] LustreError: 149270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5948.643222] LustreError: 149270:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 5953.746443] LustreError: 149271:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5953.783445] LustreError: 149271:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5955.391756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5955.482905] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5958.881900] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5960.689542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5960.703881] Lustre: Skipped 1 previous similar message [ 5960.719882] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5960.807906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5962.461659] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5976.042348] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5976.070926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5976.088491] Lustre: Skipped 1 previous similar message [ 5976.120177] Lustre: Skipped 1 previous similar message [ 5981.178901] LustreError: 152473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5981.224098] LustreError: 152473:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5981.412476] Lustre: server umount lustre-OST0000 complete [ 5991.396886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5991.410818] Lustre: Skipped 1 previous similar message [ 5991.411618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5991.422073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5991.438300] Lustre: Skipped 1 previous similar message [ 5996.514091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6001.633799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6001.647711] Lustre: Skipped 2 previous similar messages [ 6002.143120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6002.393350] Lustre: server umount lustre-MDT0000 complete [ 6005.729201] LustreError: 151244:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6005.770106] LustreError: 151244:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6008.024858] LustreError: 149255:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785394280 with bad export cookie 15002420718516218510 [ 6008.042707] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6008.042977] LustreError: 149255:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6009.297424] Lustre: server umount lustre-MDT0001 complete [ 6025.838359] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 6030.022842] Key type lgssc unregistered [ 6030.564241] LNet: 154897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6030.580957] LNetError: 154897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6030.605111] LNet: Removed LNI 192.168.203.101@tcp [ 6031.917167] Key type .llcrypt unregistered [ 6031.924100] Key type ._llcrypt unregistered [ 6085.370429] Key type ._llcrypt registered [ 6085.388022] Key type .llcrypt registered [ 6085.598896] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 6087.787187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6087.795341] alg: No test for adler32 (adler32-zlib) [ 6089.112508] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 6089.340643] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 6091.007168] Key type lgssc registered [ 6092.422568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6105.992659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6106.045090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6108.025622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6114.121284] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6117.368269] LustreError: 156422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6117.406377] LustreError: 156422:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6122.468897] LustreError: 156423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6127.587902] LustreError: 156422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6127.936155] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6132.705321] LustreError: 156423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6133.955572] LustreError: 156451:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6133.973605] LustreError: 156451:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6133.989639] LustreError: 156451:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 26, retries 0, failed: rc = -5 [ 6134.653744] Lustre: server umount lustre-MDT0000 complete [ 6154.393994] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 6165.991562] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 6214.892954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6215.271464] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6215.367084] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6215.570037] Lustre: lustre-MDT0000: new disk, initializing [ 6215.762918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6215.843355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6225.189925] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6244.075835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6244.367139] Lustre: 159760:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6244.502315] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6244.506570] Lustre: Skipped 1 previous similar message [ 6244.681237] Lustre: lustre-MDT0001: new disk, initializing [ 6244.813758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6244.907577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6244.930038] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6253.151042] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6268.844609] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6277.184197] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6288.500620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6288.917952] Lustre: lustre-OST0000: new disk, initializing [ 6288.922189] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6288.944174] Lustre: 161729:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6289.062775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6297.319992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6297.360039] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6297.725884] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6301.047263] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6313.713791] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6319.252889] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 6321.385791] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:56:32 (1785394592) [ 6630.370142] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6630.374284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6630.402203] Lustre: Skipped 2 previous similar messages [ 6630.416855] Lustre: Skipped 2 previous similar messages [ 6635.445202] Lustre: server umount lustre-MDT0000 complete [ 6635.489973] LustreError: 163496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6641.146964] LustreError: 159751:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785394913 with bad export cookie 11792659009129211350 [ 6641.163296] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6641.164465] LustreError: 159751:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6645.728108] LustreError: 159767:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6645.729264] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6645.747439] LustreError: 159767:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6645.759506] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6648.074234] Lustre: server umount lustre-MDT0001 complete [ 6659.871660] Lustre: server umount lustre-OST0000 complete [ 6671.976801] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 6683.423544] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 6732.046385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6732.410737] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6732.482481] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6732.641125] Lustre: lustre-MDT0000: new disk, initializing [ 6732.808123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6732.902164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6740.626881] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6754.677580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6754.861582] Lustre: 166417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6754.962768] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6754.969025] Lustre: Skipped 1 previous similar message [ 6755.134966] Lustre: lustre-MDT0001: new disk, initializing [ 6755.320762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6755.408527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6755.428217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6761.179363] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6776.250605] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6786.192675] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6795.581538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6795.818138] Lustre: lustre-OST0000: new disk, initializing [ 6795.821245] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6795.826421] Lustre: 168387:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6795.923849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6804.609988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6804.623648] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6804.777852] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6804.945487] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6816.695474] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6823.047922] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6823.275936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6828.788983] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6829.103976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6835.694184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6835.713978] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6835.715483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6835.715487] Lustre: Skipped 1 previous similar message [ 6835.749363] Lustre: Skipped 1 previous similar message [ 6835.795573] LustreError: Skipped 1 previous similar message [ 6838.164959] Lustre: server umount lustre-OST0000 complete [ 6845.923805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6845.932229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6845.946990] Lustre: Skipped 1 previous similar message [ 6845.965632] Lustre: Skipped 2 previous similar messages [ 6849.474531] Lustre: server umount lustre-MDT0000 complete [ 6851.040394] LustreError: 167788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6851.065976] LustreError: 167788:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6854.642406] LustreError: 166409:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785395127 with bad export cookie 11792659009129370187 [ 6854.655200] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6854.674236] LustreError: 166409:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6855.587356] Lustre: server umount lustre-MDT0001 complete [ 6866.753488] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:05:37 (1785395137) [ 6888.337631] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 6900.278280] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 6948.540411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6948.857319] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6948.952819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6949.068974] Lustre: lustre-MDT0000: new disk, initializing [ 6949.259306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6949.283577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6954.978245] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6967.435207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6967.644141] Lustre: 173147:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6968.081978] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6975.068385] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6989.257262] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6996.769060] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7005.259334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7005.765182] Lustre: lustre-OST0000: new disk, initializing [ 7005.769876] Lustre: Skipped 1 previous similar message [ 7005.777654] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7005.782633] Lustre: Skipped 2 previous similar messages [ 7005.794769] Lustre: 175130:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7012.404022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7012.420129] Lustre: Skipped 1 previous similar message [ 7012.432741] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7012.579822] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7013.206722] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7025.153861] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7038.431781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7038.440186] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7038.441636] LustreError: Skipped 1 previous similar message [ 7038.442922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7038.473075] Lustre: Skipped 1 previous similar message [ 7038.503616] Lustre: Skipped 1 previous similar message [ 7040.703948] Lustre: server umount lustre-OST0000 complete [ 7048.699858] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7048.702138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7048.726411] Lustre: Skipped 1 previous similar message [ 7048.759776] Lustre: Skipped 1 previous similar message [ 7052.376274] Lustre: server umount lustre-MDT0000 complete [ 7053.792258] LustreError: 173158:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7053.812190] LustreError: 173158:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7057.355343] LustreError: 173139:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785395330 with bad export cookie 11792659009129371069 [ 7057.360540] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7057.367102] LustreError: 173139:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7057.962392] Lustre: server umount lustre-MDT0001 complete [ 7068.067942] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:08:59 (1785395339) [ 7093.263215] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 7109.570832] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 7169.479881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7169.915790] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7169.940181] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7170.043472] Lustre: lustre-MDT0000: new disk, initializing [ 7170.125771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7170.128301] Lustre: Skipped 2 previous similar messages [ 7170.159371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7176.405820] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7190.278471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7190.482378] Lustre: 179478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7191.038854] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7197.981959] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7211.095780] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7219.547967] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7227.518426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7227.959846] Lustre: 181460:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7234.139785] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7234.311775] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7235.627422] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7247.419608] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7260.136584] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7260.145602] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7260.149961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7260.161856] Lustre: Skipped 1 previous similar message [ 7260.205642] LustreError: Skipped 1 previous similar message [ 7262.627448] Lustre: server umount lustre-OST0000 complete [ 7267.809422] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7273.909618] Lustre: server umount lustre-MDT0000 complete [ 7275.489842] LustreError: 179490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7275.528384] LustreError: 179490:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7279.466017] LustreError: 180291:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785395552 with bad export cookie 11792659009129372154 [ 7279.472508] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7279.490231] LustreError: 180291:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7280.365963] Lustre: server umount lustre-MDT0001 complete [ 7295.682230] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 7301.862760] Key type lgssc unregistered [ 7302.556472] LNet: 183319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7302.560250] LNetError: 183319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7302.600620] LNet: Removed LNI 192.168.203.101@tcp [ 7304.768391] Key type .llcrypt unregistered [ 7304.779389] Key type ._llcrypt unregistered [ 7344.637296] Key type ._llcrypt registered [ 7344.648341] Key type .llcrypt registered [ 7344.904796] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 7346.496877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7346.743200] alg: No test for adler32 (adler32-zlib) [ 7347.934395] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 7348.504094] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 7350.416793] Key type lgssc registered [ 7352.377175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7365.298847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7365.346336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7366.895389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7374.663822] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7376.359088] LustreError: 184741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7376.385644] LustreError: 184741:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7381.487764] LustreError: 184742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7386.599665] LustreError: 184741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7387.908955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7388.763584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7396.552995] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7406.888814] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7415.144470] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7424.109801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7424.687685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7433.283980] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7437.054180] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 7443.235643] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7464.928690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7464.934581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7464.955321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7468.000390] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7468.008491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7468.023617] Lustre: Skipped 1 previous similar message [ 7469.455894] Lustre: server umount lustre-OST0000 complete [ 7475.683908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7475.694483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7475.720969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7478.248387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7478.253475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7480.557391] Lustre: server umount lustre-MDT0000 complete [ 7483.374227] LustreError: 185747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7483.413312] LustreError: 185747:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7485.885578] LustreError: 184723:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785395758 with bad export cookie 6355803408951951306 [ 7485.892184] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7485.902424] LustreError: 184723:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7486.533355] Lustre: server umount lustre-MDT0001 complete [ 7507.798138] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 7511.939915] Key type lgssc unregistered [ 7512.504649] LNet: 188325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7512.522140] LNetError: 188325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7512.549698] LNet: Removed LNI 192.168.203.101@tcp [ 7513.692131] Key type .llcrypt unregistered [ 7513.697119] Key type ._llcrypt unregistered [ 7550.448426] Key type ._llcrypt registered [ 7550.454668] Key type .llcrypt registered [ 7550.668826] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 7552.473341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7552.514047] alg: No test for adler32 (adler32-zlib) [ 7553.893755] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 7554.582926] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 7556.631226] Key type lgssc registered [ 7559.533656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7571.963400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7572.000381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7574.006042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7579.202935] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7584.225124] LustreError: 189744:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7584.265375] LustreError: 189744:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7589.344360] LustreError: 189745:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7591.363928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7592.243977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7598.411671] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7607.947699] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7614.852801] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7624.021464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7624.397462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7632.604304] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7643.303726] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7643.903414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7652.349144] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7652.367048] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7652.393175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7654.376404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7654.393532] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7654.406430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7655.680524] Lustre: lustre-OST0000: Not available for connect from 192.168.203.1@tcp (stopping) [ 7655.683255] Lustre: Skipped 1 previous similar message [ 7659.495322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7664.608040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7664.623294] Lustre: Skipped 2 previous similar messages [ 7665.119119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7665.423939] Lustre: server umount lustre-OST0000 complete [ 7674.903889] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff93ed09861000.ost_server_uuid 50 [ 7677.078183] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 7692.679747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7692.688039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7692.706527] Lustre: Skipped 1 previous similar message [ 7692.733262] Lustre: Skipped 1 previous similar message [ 7699.255616] Lustre: server umount lustre-MDT0000 complete [ 7699.951950] LustreError: 189741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7699.993498] LustreError: 189741:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7705.063440] LustreError: 190474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7705.097678] LustreError: 190474:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7707.819550] LustreError: 190481:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785395980 with bad export cookie 2096499417192057144 [ 7707.845180] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7707.847723] LustreError: 190481:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7708.560901] Lustre: server umount lustre-MDT0001 complete [ 7720.445537] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 03:19:51 (1785395991) [ 7729.795085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7730.625417] LustreError: 193669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7730.727095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7737.939356] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7740.900346] LustreError: 193670:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7740.910698] LustreError: 193670:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7750.909350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7751.795835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7758.198369] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7769.540344] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7776.124683] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7784.981893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7785.452188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7794.697550] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7801.862855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7805.451180] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7813.923676] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7814.310712] Lustre: lustre-OST0001: new disk, initializing [ 7814.320862] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7814.337790] Lustre: 196515:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7814.517720] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7820.458022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7820.479372] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7820.616307] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7823.240534] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7835.522394] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7841.255184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7841.262832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7841.269386] Lustre: Skipped 1 previous similar message [ 7841.281954] Lustre: Skipped 3 previous similar messages [ 7845.461700] Lustre: server umount lustre-OST0000 complete [ 7846.379582] LustreError: 195492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7846.407747] LustreError: 195492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7854.100855] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7854.315690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7860.706941] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7861.049610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7877.088928] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7877.092677] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7877.112184] LustreError: Skipped 1 previous similar message [ 7877.121626] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7877.121633] Lustre: Skipped 1 previous similar message [ 7877.163892] Lustre: Skipped 1 previous similar message [ 7882.212564] LustreError: 196786:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7882.248109] LustreError: 196786:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 7883.665565] Lustre: server umount lustre-OST0001 complete [ 7889.888145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7889.893486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7896.269162] Lustre: server umount lustre-MDT0000 complete [ 7901.459339] LustreError: 195514:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785396174 with bad export cookie 2096499417192058208 [ 7901.477426] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7901.777442] Lustre: server umount lustre-MDT0001 complete [ 7930.855241] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 03:23:21 (1785396201) [ 7939.700320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7940.290709] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7940.402401] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7940.719553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7947.293725] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7958.830517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7959.050622] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7959.132384] Lustre: 199957:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7967.044838] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7979.161289] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7989.788524] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8000.013594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8000.176483] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8008.432451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 8011.369756] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8022.100046] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8034.550635] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8034.722516] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8035.195282] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8035.212975] Lustre: Skipped 2 previous similar messages [ 8042.936647] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 8047.088227] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8058.858719] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8064.992236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8064.996631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8065.006328] LustreError: Skipped 1 previous similar message [ 8065.007783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8065.007787] Lustre: Skipped 6 previous similar messages [ 8065.113429] Lustre: Skipped 2 previous similar messages [ 8070.344115] LustreError: 201813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8070.365546] LustreError: 201813:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8070.556933] Lustre: server umount lustre-OST0000 complete [ 8092.221746] Lustre: server umount lustre-OST0001 complete [ 8097.760588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8097.772143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8106.879207] Lustre: server umount lustre-MDT0000 complete [ 8109.536813] Lustre: 189057:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785396366/real 1785396366] req@000000009a7ba57a x1872123214881152/t0(0) o13->lustre-OST0001-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785396382 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8110.111092] Lustre: 189056:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785396366/real 1785396366] req@00000000ce1c67aa x1872123214881024/t0(0) o400->lustre-OST0001-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785396382 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8111.681400] LustreError: 202848:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785396384 with bad export cookie 2096499417192059622 [ 8111.681841] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8111.692476] LustreError: 202848:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8112.435926] Lustre: server umount lustre-MDT0001 complete [ 8143.952459] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 03:26:54 (1785396414) [ 8165.094504] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 8178.106669] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 8240.768080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8241.396263] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8241.435527] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8241.713395] Lustre: lustre-MDT0000: new disk, initializing [ 8241.806113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8241.847129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8249.241708] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8262.575090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8262.784248] Lustre: 208069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8262.886114] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8262.892553] Lustre: Skipped 1 previous similar message [ 8263.065234] Lustre: lustre-MDT0001: new disk, initializing [ 8263.274596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8263.286311] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8270.386627] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8283.699851] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8291.396645] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8300.906696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8301.406805] Lustre: lustre-OST0000: new disk, initializing [ 8301.417219] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8301.430195] Lustre: 210038:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8309.859463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8309.875249] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8310.076413] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8310.712199] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8322.462442] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8328.593044] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8328.858499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8334.841420] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8335.245250] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8340.966095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8340.969364] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8340.973806] LustreError: Skipped 1 previous similar message [ 8340.977770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8340.977777] Lustre: Skipped 9 previous similar messages [ 8341.030500] Lustre: Skipped 3 previous similar messages [ 8344.071653] Lustre: server umount lustre-OST0000 complete [ 8355.700651] Lustre: server umount lustre-MDT0000 complete [ 8358.369413] LustreError: 209632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8358.389877] LustreError: 209632:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 8360.462492] LustreError: 208060:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785396633 with bad export cookie 2096499417192061407 [ 8360.484683] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8361.444874] Lustre: server umount lustre-MDT0001 complete [ 8372.813969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8380.137744] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8391.560704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8398.976784] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8409.552647] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8417.849283] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8426.060736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8434.815283] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8446.172038] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8453.477866] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8453.976704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8460.947818] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8461.204732] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8470.095787] Lustre: server umount lustre-OST0000 complete [ 8478.530526] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8485.523431] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8519.787296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8520.312257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8520.322665] Lustre: Skipped 5 previous similar messages [ 8530.531219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8530.552435] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8530.571398] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8530.595644] Lustre: Skipped 1 previous similar message [ 8532.172794] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8544.756205] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8547.769344] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8556.512724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8556.517422] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8556.528034] Lustre: Skipped 4 previous similar messages [ 8556.552643] LustreError: Skipped 1 previous similar message [ 8560.639783] Lustre: server umount lustre-OST0000 complete [ 8573.471814] Lustre: server umount lustre-MDT0000 complete [ 8577.765394] LustreError: 212239:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785396850 with bad export cookie 2096499417192062289 [ 8577.771444] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8577.776480] LustreError: 212239:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8579.312888] Lustre: server umount lustre-MDT0001 complete [ 8591.597190] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 03:34:21 (1785396861) [ 8600.489335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8608.976612] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8616.928288] LustreError: 218235:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8616.952497] LustreError: 218235:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 8619.594606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8628.084573] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8637.676450] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8644.396451] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8653.086410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8662.883500] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8674.803362] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8681.811608] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8682.086136] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8687.272842] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8687.672729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8696.959124] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8697.310365] Lustre: lustre-OST0001: new disk, initializing [ 8697.318969] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8697.331776] Lustre: 221607:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8705.931576] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8706.664586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8706.678893] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8706.828928] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8717.274524] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8723.702518] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 8724.025158] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8731.222955] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 8731.557812] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8737.768576] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8737.772465] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8737.782301] Lustre: Skipped 11 previous similar messages [ 8737.801370] LustreError: Skipped 2 previous similar messages [ 8740.806820] Lustre: server umount lustre-OST0001 complete [ 8749.275227] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 8749.728571] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 8756.944954] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 8757.251128] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 8788.948906] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8794.615761] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8794.646198] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8794.658101] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8794.683480] Lustre: Skipped 1 previous similar message [ 8797.967385] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8807.201418] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8809.088885] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8815.074540] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8815.099530] Lustre: Skipped 6 previous similar messages [ 8817.611384] Lustre: server umount lustre-OST0001 complete [ 8845.505082] Lustre: server umount lustre-OST0000 complete [ 8857.080831] Lustre: server umount lustre-MDT0000 complete [ 8861.987201] LustreError: 218216:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785397134 with bad export cookie 2096499417192063255 [ 8861.987766] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8862.000357] LustreError: 218216:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8868.607636] Lustre: server umount lustre-MDT0001 complete [ 8906.198667] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 03:39:35 (1785397175) [ 8918.294724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8918.802075] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8918.875471] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8926.502798] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8942.796594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8943.171857] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8943.254155] Lustre: 226855:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8952.540629] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8965.522321] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8977.428847] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8987.028542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8987.354975] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8995.305174] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9005.646233] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9018.374449] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9018.466846] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9029.267663] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9040.313577] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9042.379095] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9047.893882] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 9048.290968] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9055.735044] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 9056.087422] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9066.164687] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93ed1105b800.ost_server_uuid 50 [ 9068.528141] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff93ed1105b800.ost_server_uuid in IDLE state after 0 sec [ 9069.905704] Lustre: Permanently deactivating lustre-OST0001 [ 9069.946325] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 9075.566271] Lustre: Permanently reactivating lustre-OST0001 [ 9075.617954] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 9075.626865] Lustre: Skipped 2 previous similar messages [ 9087.466414] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9087.480698] LustreError: Skipped 5 previous similar messages [ 9089.888062] Lustre: server umount lustre-OST0001 complete [ 9101.330876] Lustre: server umount lustre-OST0000 complete [ 9113.449362] Lustre: server umount lustre-MDT0000 complete [ 9119.469653] LustreError: 226848:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785397392 with bad export cookie 2096499417192064543 [ 9119.487737] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9119.492107] LustreError: 226848:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9120.147681] Lustre: server umount lustre-MDT0001 complete [ 9154.405866] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 03:43:44 (1785397424) [ 9163.990583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9164.435971] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9164.508070] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9164.521428] Lustre: Skipped 2 previous similar messages [ 9164.820205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9164.823147] Lustre: Skipped 9 previous similar messages [ 9171.337058] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9185.236255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9185.403094] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9185.434099] Lustre: 233563:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9192.792893] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9203.076698] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9211.453663] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9221.386432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9221.611510] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9221.626483] Lustre: Permanently deactivating lustre-OST0000 [ 9221.637732] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 9227.302186] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9229.812161] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9240.027427] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9249.685943] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9249.811576] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9258.826856] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 9259.126941] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9272.635869] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9276.660458] Lustre: Permanently reactivating lustre-OST0000 [ 9276.715143] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 9276.726708] Lustre: Skipped 2 previous similar messages [ 9283.165227] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 9283.184555] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 55 seconds. I think it's dead, and I am evicting it. exp 000000007985674d, cur 1785397555 deadline 1785397530 last 1785397500 [ 9283.240657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9299.955200] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9299.969197] Lustre: Skipped 17 previous similar messages [ 9302.601381] Lustre: server umount lustre-OST0001 complete [ 9305.073890] LustreError: 235415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9305.102085] LustreError: 235415:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 47 previous similar messages [ 9314.006744] Lustre: server umount lustre-OST0000 complete [ 9325.881257] Lustre: server umount lustre-MDT0000 complete [ 9331.905981] LustreError: 233555:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785397604 with bad export cookie 2096499417192066216 [ 9331.919441] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9331.929625] LustreError: 233555:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9333.134621] Lustre: server umount lustre-MDT0001 complete [ 9352.283383] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 9356.743324] Key type lgssc unregistered [ 9357.104953] LNet: 238468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9357.114357] LNetError: 238468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9357.139190] LNet: Removed LNI 192.168.203.101@tcp [ 9358.823624] Key type .llcrypt unregistered [ 9358.825220] Key type ._llcrypt unregistered [ 9395.874430] Key type ._llcrypt registered [ 9395.877963] Key type .llcrypt registered [ 9396.078185] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 9397.828433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9398.175083] alg: No test for adler32 (adler32-zlib) [ 9399.362967] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 9399.732234] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 9401.519163] Key type lgssc registered [ 9402.554616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9417.765901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9417.827294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9420.209471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9427.342388] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9429.496569] LustreError: 239939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9429.544720] LustreError: 239939:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9434.593355] LustreError: 239938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9437.857379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9438.034643] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9438.068502] Lustre: 239927:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9438.080438] Lustre: Permanently deactivating lustre-MDT0001 [ 9438.094130] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 9438.552401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9444.395857] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 9445.315934] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9455.517894] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9463.411815] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9473.396517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9474.216144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9483.446812] LustreError: 242087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9483.484938] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 9484.778276] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9488.873633] LustreError: 242087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9490.956251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 9494.721311] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9496.033671] LustreError: 241780:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9496.080399] LustreError: 241780:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9503.799206] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9504.201958] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9509.381327] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 9509.381506] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 9512.582647] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9523.299952] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9528.533143] Lustre: Permanently reactivating lustre-MDT0001 [ 9528.561426] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 9528.566286] Lustre: Skipped 2 previous similar messages [ 9533.159369] Lustre: lustre-MDT0001: Client 8c817622-3782-439b-8684-95e18e2e81c9 (at 192.168.203.1@tcp) reconnecting [ 9538.635045] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9538.646993] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 9538.654904] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 9538.662813] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 9542.739420] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9545.444244] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9554.980584] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 9555.413544] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 9564.348231] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 9564.811493] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 9568.975880] Lustre: Permanently deactivating lustre-MDT0001 [ 9569.028222] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 9569.036381] Lustre: Skipped 2 previous similar messages [ 9577.336245] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 9586.156028] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9586.172388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9586.181021] Lustre: Skipped 3 previous similar messages [ 9586.197073] Lustre: Skipped 3 previous similar messages [ 9591.264515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9591.279227] Lustre: Skipped 3 previous similar messages [ 9592.103740] Lustre: server umount lustre-MDT0000 complete [ 9596.389407] LustreError: 239935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9596.413703] LustreError: 239935:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9597.675552] LustreError: 243492:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785397870 with bad export cookie 8871793497792450017 [ 9597.678887] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9597.690963] LustreError: 243492:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9598.589314] Lustre: server umount lustre-MDT0001 complete [ 9604.523576] Lustre: server umount lustre-OST0000 complete [ 9620.657927] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 03:51:31 (1785397891) [ 9635.817318] Lustre: server umount lustre-OST0001 complete [ 9646.451972] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 9663.120312] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 9714.511593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9714.899587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9714.910825] Lustre: Skipped 2 previous similar messages [ 9714.981150] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9715.122830] Lustre: lustre-MDT0000: new disk, initializing [ 9715.262262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9715.301064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9721.802529] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9736.168584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9736.425067] Lustre: 248230:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9736.539758] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9736.550467] Lustre: Skipped 1 previous similar message [ 9736.716993] Lustre: lustre-MDT0001: new disk, initializing [ 9736.904792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9736.961451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9736.981144] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9743.529414] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9757.124165] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9766.336829] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9774.641750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9775.161810] Lustre: lustre-OST0000: new disk, initializing [ 9775.171619] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9775.195359] Lustre: 250200:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9775.349120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9783.362500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9783.379128] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9783.465869] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9783.534783] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9797.423888] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9800.828204] LustreError: 249853:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 9810.914238] LustreError: 249853:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 9810.950199] LustreError: 248237:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 9813.653694] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9813.896536] LustreError: 250184:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9813.918706] Lustre: lustre-OST0001: new disk, initializing [ 9813.935240] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9813.951852] Lustre: 251317:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9814.082673] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9819.779522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9819.807718] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9819.971279] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9820.965685] LustreError: 248237:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 9824.001986] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9836.127428] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9838.691566] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9845.734548] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9845.737296] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9845.750864] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9845.790637] Lustre: Skipped 1 previous similar message [ 9848.072746] Lustre: server umount lustre-OST0001 complete [ 9850.853732] LustreError: 250185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9850.922775] LustreError: 250185:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9875.937889] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9875.959241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9875.999183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9876.016373] Lustre: Skipped 2 previous similar messages [ 9878.881880] Lustre: server umount lustre-OST0000 complete [ 9885.665455] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9885.671721] LustreError: Skipped 1 previous similar message [ 9885.678161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9885.694239] Lustre: Skipped 1 previous similar message [ 9885.704676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9885.718706] Lustre: Skipped 2 previous similar messages [ 9892.250408] Lustre: server umount lustre-MDT0000 complete [ 9897.308124] LustreError: 248223:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785398169 with bad export cookie 8871793497792452824 [ 9897.310353] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9897.319313] LustreError: 248223:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9904.083659] Lustre: server umount lustre-MDT0001 complete [ 9919.894865] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 9924.162918] Key type lgssc unregistered [ 9924.488646] LNet: 253353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9924.497935] LNetError: 253353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9924.522238] LNet: Removed LNI 192.168.203.101@tcp [ 9926.044611] Key type .llcrypt unregistered [ 9926.046045] Key type ._llcrypt unregistered [ 9991.795579] Key type ._llcrypt registered [ 9991.798306] Key type .llcrypt registered [ 9992.049554] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 9993.818700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9993.864693] alg: No test for adler32 (adler32-zlib) [ 9995.083257] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 9995.547204] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 9997.431190] Key type lgssc registered [ 9998.915475] Lustre: Echo OBD driver; http://www.lustre.org/ [10010.265113] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10010.313420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10011.835572] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10011.870470] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10012.213820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10019.247952] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10031.220334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10031.331211] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10031.362425] Lustre: 255568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10031.708284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10037.719949] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10049.110458] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10059.268854] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10069.340098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10069.789608] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10069.973129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10075.830960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [10078.145273] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10087.711329] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10152.416343] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10152.429045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10152.466032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10156.317408] Lustre: server umount lustre-OST0000 complete [10162.359986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [10180.293111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10181.120257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10189.435233] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10192.443112] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [10192.469571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [10192.470550] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [10199.677356] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10246.538363] Lustre: server umount lustre-OST0000 complete [10251.767440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10251.783731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10251.805724] Lustre: Skipped 1 previous similar message [10251.820401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10251.834793] Lustre: Skipped 1 previous similar message [10253.810830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10253.832807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10258.913371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10258.924365] Lustre: Skipped 2 previous similar messages [10261.084391] Lustre: server umount lustre-MDT0000 complete [10265.055145] Lustre: 254892:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785398521/real 1785398521] req@00000000920b0d40 x1872125774349568/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785398537 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10265.059529] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10265.091610] Lustre: 254892:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10266.554459] LustreError: 257428:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785398539 with bad export cookie 6179200194658834261 [10266.561052] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10266.571561] LustreError: 257428:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 1 previous similar message [10267.495633] Lustre: server umount lustre-MDT0001 complete [10285.009781] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [10290.887341] Key type lgssc unregistered [10291.620318] LNet: 261028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10291.640334] LNetError: 261028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10291.691344] LNet: Removed LNI 192.168.203.101@tcp [10293.953687] Key type .llcrypt unregistered [10293.956933] Key type ._llcrypt unregistered [10331.337957] Key type ._llcrypt registered [10331.342655] Key type .llcrypt registered [10331.496631] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [10333.031372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10333.194880] alg: No test for adler32 (adler32-zlib) [10334.388728] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [10334.610119] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [10336.329912] Key type lgssc registered [10338.331844] Lustre: Echo OBD driver; http://www.lustre.org/ [10349.633536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10349.671282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10351.513393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10359.014957] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10361.830356] LustreError: 262451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10361.853259] LustreError: 262451:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10366.957802] LustreError: 262450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10370.265019] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10370.964115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10377.671083] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10388.065971] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10396.654284] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10405.899054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10406.318112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10415.347471] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10421.699778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [10424.911348] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10456.544181] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10456.558460] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10456.608868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10457.569610] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10457.573059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10457.574685] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10457.596468] Lustre: Skipped 1 previous similar message [10461.279229] Lustre: server umount lustre-OST0000 complete [10469.533820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10469.587781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10474.980243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10475.002303] Lustre: Skipped 2 previous similar messages [10476.412767] Lustre: server umount lustre-MDT0000 complete [10480.101051] LustreError: 262445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10480.154205] LustreError: 262445:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10483.202697] LustreError: 262450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10483.258730] LustreError: 262450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10484.840031] LustreError: 262431:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785398757 with bad export cookie 4956579962410689861 [10484.842534] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10484.872729] LustreError: 262431:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [10486.278317] Lustre: server umount lustre-MDT0001 complete [10501.544118] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [10504.882299] Key type lgssc unregistered [10505.361446] LNet: 266762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10505.369923] LNetError: 266762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10505.404977] LNet: Removed LNI 192.168.203.101@tcp [10507.136147] Key type .llcrypt unregistered [10507.141539] Key type ._llcrypt unregistered [10539.086933] Key type ._llcrypt registered [10539.093708] Key type .llcrypt registered [10539.373456] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [10540.959246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10541.213682] alg: No test for adler32 (adler32-zlib) [10543.471891] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [10544.365850] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [10546.391845] Key type lgssc registered [10549.692479] Lustre: Echo OBD driver; http://www.lustre.org/ [10562.949963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10562.966153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10564.452153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10571.499911] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10574.826275] LustreError: 267986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10574.862834] LustreError: 267986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10579.945113] LustreError: 267987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10585.079415] LustreError: 267986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10585.504697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10586.071937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10591.697292] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10603.836593] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10614.028728] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10622.467789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10622.865479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10632.104506] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10635.194539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10642.692448] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10660.843268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10660.846299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10660.846849] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10660.868770] Lustre: Skipped 1 previous similar message [10660.883369] Lustre: Skipped 1 previous similar message [10665.350639] Lustre: server umount lustre-OST0000 complete [10671.072943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10671.101995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10676.194483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10676.212661] Lustre: Skipped 2 previous similar messages [10681.313622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10681.326980] Lustre: Skipped 1 previous similar message [10685.407142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10686.028542] Lustre: server umount lustre-MDT0000 complete [10686.436118] LustreError: 267986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10686.452754] LustreError: 267986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10691.559591] LustreError: 267983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10691.594104] LustreError: 267983:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10691.641439] LustreError: 267968:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785398964 with bad export cookie 1330182398649912339 [10691.663529] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10692.784116] Lustre: server umount lustre-MDT0001 complete [10714.993676] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [10721.563339] Key type lgssc unregistered [10722.273937] LNet: 271693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10722.289026] LNetError: 271693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10722.334493] LNet: Removed LNI 192.168.203.101@tcp [10724.015585] Key type .llcrypt unregistered [10724.017103] Key type ._llcrypt unregistered [10759.498200] Key type ._llcrypt registered [10759.500775] Key type .llcrypt registered [10759.668473] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [10761.805112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10762.142944] alg: No test for adler32 (adler32-zlib) [10763.313169] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [10763.892393] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [10765.720149] Key type lgssc registered [10768.597441] Lustre: Echo OBD driver; http://www.lustre.org/ [10779.274333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10779.300520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10781.132366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10787.469456] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10790.398246] LustreError: 273112:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10790.429637] LustreError: 273112:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10795.504897] LustreError: 273113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10799.233612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10799.877727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10807.792485] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10818.145609] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10824.787568] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10833.846825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10834.865957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10847.043798] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10851.357438] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [10860.129737] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10896.352702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10896.368801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10896.387605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10897.377086] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10897.407617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10897.438145] Lustre: Skipped 1 previous similar message [10899.635498] Lustre: server umount lustre-OST0000 complete [10907.627260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10907.643205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10907.643330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10907.643518] Lustre: Skipped 1 previous similar message [10907.690394] Lustre: Skipped 1 previous similar message [10910.911119] Lustre: server umount lustre-MDT0000 complete [10912.739311] LustreError: 273840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10915.505537] LustreError: 273092:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785399188 with bad export cookie 8146127168020409811 [10915.509286] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10915.521675] LustreError: 273092:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 1 previous similar message [10916.507432] Lustre: server umount lustre-MDT0001 complete [10931.733584] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [10936.173508] Key type lgssc unregistered [10936.739560] LNet: 277424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10936.756261] LNetError: 277424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10937.766615] LNet: Removed LNI 192.168.203.101@tcp [10939.288925] Key type .llcrypt unregistered [10939.290648] Key type ._llcrypt unregistered [10970.562858] Key type ._llcrypt registered [10970.566984] Key type .llcrypt registered [10970.739962] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [10972.039473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10972.158073] alg: No test for adler32 (adler32-zlib) [10973.510925] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [10973.876859] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [10975.647136] Key type lgssc registered [10977.615850] Lustre: Echo OBD driver; http://www.lustre.org/ [10988.402761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10988.447201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10990.116300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10995.438752] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11000.290951] LustreError: 278685:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11000.317242] LustreError: 278685:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11005.410419] LustreError: 278687:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11006.718196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11007.330401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11014.753906] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11026.407424] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11034.292762] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11043.514144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11043.795648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11052.630929] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11060.230854] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [11062.465948] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11080.674502] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11080.676229] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11080.698405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11084.960866] Lustre: server umount lustre-OST0000 complete [11090.921453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11090.958075] Lustre: Skipped 1 previous similar message [11090.977141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11090.979476] Lustre: Skipped 1 previous similar message [11096.033120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11097.225275] Lustre: server umount lustre-MDT0000 complete [11101.153860] LustreError: 278673:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11101.175055] LustreError: 278673:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11102.265800] LustreError: 280553:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785399374 with bad export cookie 13585857231698901284 [11102.270430] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11102.277388] LustreError: 280553:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [11103.152773] Lustre: server umount lustre-MDT0001 complete [11117.078044] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [11121.047940] Key type lgssc unregistered [11121.445590] LNet: 282390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11121.462627] LNetError: 282390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11121.474566] LNet: Removed LNI 192.168.203.101@tcp [11122.577890] Key type .llcrypt unregistered [11122.581928] Key type ._llcrypt unregistered [11138.531265] Key type ._llcrypt registered [11138.532929] Key type .llcrypt registered [11138.720423] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [11159.060015] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [11181.390599] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [11184.089940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11184.217163] alg: No test for adler32 (adler32-zlib) [11186.881878] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [11187.788445] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [11189.807127] Key type lgssc registered [11193.739414] Lustre: Echo OBD driver; http://www.lustre.org/ [11251.770042] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11251.837668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11253.502236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11253.576432] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11253.811040] Lustre: lustre-MDT0000: new disk, initializing [11253.985355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11254.026502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11261.305492] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11275.355041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11275.468725] Lustre: 285948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11275.522319] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11275.527567] Lustre: Skipped 1 previous similar message [11275.665354] Lustre: lustre-MDT0001: new disk, initializing [11275.805996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11275.849619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11275.860073] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11281.447624] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11293.975049] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11303.749384] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11314.436219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11315.031934] Lustre: lustre-OST0000: new disk, initializing [11315.039314] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11315.046145] Lustre: 287918:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11315.184853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11321.510215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11321.535684] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11321.812626] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11326.655564] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11338.693494] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11346.539575] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11346.843073] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11352.894947] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11353.204601] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11357.679284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11357.683250] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11357.695487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11357.724937] LustreError: Skipped 1 previous similar message [11362.061961] Lustre: server umount lustre-OST0000 complete [11367.906972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11367.910027] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11367.918694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11367.923798] Lustre: Skipped 2 previous similar messages [11367.934935] Lustre: Skipped 2 previous similar messages [11373.027885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11378.148926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11378.167234] Lustre: Skipped 2 previous similar messages [11382.239269] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11383.031289] Lustre: server umount lustre-MDT0000 complete [11388.393449] LustreError: 285959:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11388.438783] LustreError: 285959:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11390.515697] LustreError: 285938:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785399663 with bad export cookie 7636079736547842506 [11390.539724] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11390.553129] LustreError: 285938:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 3 previous similar messages [11391.643791] Lustre: server umount lustre-MDT0001 complete [11404.107900] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 04:21:14 (1785399674) [11412.019771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11412.543084] LustreError: 290338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11412.657637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11418.087628] LustreError: 290339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11418.940606] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11423.204961] LustreError: 290338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11428.335413] LustreError: 290339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11433.614043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11434.325420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11443.009089] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11456.566844] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11463.780639] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11472.576738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11473.206719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11481.301725] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11493.027961] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11513.321125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11513.324374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11513.327783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11513.393029] Lustre: Skipped 1 previous similar message [11519.011159] Lustre: server umount lustre-OST0000 complete [11526.625824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11526.633720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11526.656511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11526.665936] Lustre: Skipped 3 previous similar messages [11531.135881] Lustre: server umount lustre-MDT0000 complete [11533.797375] LustreError: 290338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11533.821531] LustreError: 290338:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11536.695312] LustreError: 290318:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785399809 with bad export cookie 7636079736547843388 [11536.721025] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11537.217642] Lustre: server umount lustre-MDT0001 complete [11553.500979] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [11556.703933] Key type lgssc unregistered [11557.245614] LNet: 293901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11557.273607] LNetError: 293901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11558.321525] LNet: Removed LNI 192.168.203.101@tcp [11559.386595] Key type .llcrypt unregistered [11559.394042] Key type ._llcrypt unregistered [11599.043962] Key type ._llcrypt registered [11599.064575] Key type .llcrypt registered [11599.352094] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [11628.535858] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [11631.322123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11631.344093] alg: No test for adler32 (adler32-zlib) [11632.526521] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [11632.730497] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [11634.503160] Key type lgssc registered [11637.307828] Lustre: Echo OBD driver; http://www.lustre.org/ [11698.317444] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11698.387650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11699.677296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11699.781883] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11699.938117] Lustre: lustre-MDT0000: new disk, initializing [11700.060195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11700.074538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11705.950537] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11719.823379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11719.975756] Lustre: 297737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11720.112344] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11720.123744] Lustre: Skipped 1 previous similar message [11720.296649] Lustre: lustre-MDT0001: new disk, initializing [11720.464274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11720.529739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11720.551442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11727.379559] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11740.390572] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11749.569663] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11758.640727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11758.899964] Lustre: lustre-OST2710: new disk, initializing [11758.908872] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [11758.913983] Lustre: 299708:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11759.067328] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [11765.844671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [11765.864110] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [11765.975096] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [11768.070396] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11780.164055] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11788.542180] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11788.747334] Lustre: lustre-OST03e8: new disk, initializing [11788.750561] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [11788.757865] Lustre: 300772:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11788.860647] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [11794.542987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [11794.565770] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [11794.695059] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [11797.992790] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11808.344504] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11817.575410] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [11817.913027] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11824.920827] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [11825.299094] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11831.493876] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [11831.862137] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11838.959385] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [11839.286141] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11848.672794] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11848.681795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11848.697397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11851.234916] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11851.236607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11851.271038] Lustre: Skipped 2 previous similar messages [11851.293115] Lustre: Skipped 3 previous similar messages [11853.888427] Lustre: server umount lustre-MDT0000 complete [11859.315685] LustreError: 297730:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785400132 with bad export cookie 457761351734900553 [11859.321724] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11859.330479] LustreError: 297730:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [11859.722257] Lustre: server umount lustre-MDT0001 complete [11874.930823] Lustre: server umount lustre-OST2710 complete [11877.663097] Lustre: 295535:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785400134/real 1785400134] req@00000000405c27b2 x1872127490867840/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785400150 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11877.693608] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11880.927370] Lustre: 295532:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785400137/real 1785400137] req@0000000067165a78 x1872127490868096/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785400153 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11882.608524] Lustre: server umount lustre-OST03e8 complete [11906.421098] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [11920.792342] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [11973.758627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11974.120479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11974.178844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11974.313804] Lustre: lustre-MDT0000: new disk, initializing [11974.449230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11974.482204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11979.404541] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11994.144329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11994.325400] Lustre: 306031:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11994.367979] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11994.370333] Lustre: Skipped 1 previous similar message [11994.410171] Lustre: lustre-MDT0001: new disk, initializing [11994.513391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11994.587634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11994.607286] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12000.231599] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12012.297414] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12021.810734] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12034.035377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12034.340088] Lustre: lustre-OST0000: new disk, initializing [12034.345369] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12034.356482] Lustre: 308002:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12034.561301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12044.551803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12044.575746] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12044.830146] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12046.531897] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12064.595645] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12071.086833] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12071.308493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12078.173099] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12078.585278] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12085.731683] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12085.738530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12085.748508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12085.750779] Lustre: Skipped 1 previous similar message [12087.449481] Lustre: server umount lustre-OST0000 complete [12095.970116] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12095.977757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12095.992224] Lustre: Skipped 1 previous similar message [12096.021935] Lustre: Skipped 2 previous similar messages [12100.390525] Lustre: server umount lustre-MDT0000 complete [12101.088068] LustreError: 307191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12101.108853] LustreError: 307191:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12106.209456] LustreError: 306043:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12106.264863] LustreError: 306043:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12106.970072] LustreError: 306023:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785400379 with bad export cookie 457761351734902240 [12106.988103] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12107.008533] LustreError: 306023:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [12108.599972] Lustre: server umount lustre-MDT0001 complete [12124.421692] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 04:33:14 (1785400394) [12126.336292] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [12129.124589] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 04:33:19 (1785400399) [12134.362791] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [12179.419413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12179.753286] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12179.778297] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12179.918828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12185.821375] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12197.590331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12197.685497] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12197.721115] Lustre: 311867:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12204.000809] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12228.602256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12229.016283] Lustre: Denying initial registration attempt from nid 192.168.203.101@tcp, specified as failover [12229.026751] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [12229.037779] LustreError: 313321:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [12229.048540] LustreError: 313321:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [12229.064373] LustreError: 313321:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [12229.210545] Lustre: server umount lustre-OST0000 complete [12229.216216] LustreError: 313321:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [12239.328299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12239.338108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12239.356744] Lustre: Skipped 1 previous similar message [12245.400425] Lustre: server umount lustre-MDT0000 complete [12249.569564] LustreError: 311879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12249.590949] LustreError: 311879:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12250.180485] LustreError: 311859:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785400522 with bad export cookie 457761351734903129 [12250.222087] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12250.888499] Lustre: server umount lustre-MDT0001 complete [12266.411591] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [12271.404959] Key type lgssc unregistered [12271.937106] LNet: 314444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12271.948055] LNetError: 314444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12271.965918] LNet: Removed LNI 192.168.203.101@tcp [12273.353141] Key type .llcrypt unregistered [12273.358479] Key type ._llcrypt unregistered [12290.942187] Key type ._llcrypt registered [12290.945677] Key type .llcrypt registered [12291.167542] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [12292.954665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12293.029898] alg: No test for adler32 (adler32-zlib) [12294.382573] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [12294.854243] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [12296.616909] Key type lgssc registered [12297.972544] Lustre: Echo OBD driver; http://www.lustre.org/ [12364.666164] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [12375.948857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12375.992184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12377.539492] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12377.579609] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12377.876294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12385.243373] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12397.440918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12397.677318] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12397.759975] Lustre: 317216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12398.276061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12404.547694] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12413.909996] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12421.563466] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12443.956104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12444.238517] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12444.564208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12454.369410] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12464.592586] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12468.192085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12468.193531] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12468.200590] LustreError: Skipped 1 previous similar message [12468.203220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12468.240034] Lustre: Skipped 1 previous similar message [12473.619288] Lustre: server umount lustre-OST0000 complete [12478.434111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12478.452202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12478.465434] Lustre: Skipped 1 previous similar message [12478.485898] Lustre: Skipped 2 previous similar messages [12483.562356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12483.567310] Lustre: Skipped 1 previous similar message [12488.672306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12488.678981] Lustre: Skipped 1 previous similar message [12492.255285] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12492.753663] Lustre: server umount lustre-MDT0000 complete [12493.799224] LustreError: 317227:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12493.837281] LustreError: 317227:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12497.223252] LustreError: 317957:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785400769 with bad export cookie 16792053430387536771 [12497.224915] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12497.234282] LustreError: 317957:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [12497.752292] Lustre: server umount lustre-MDT0001 complete [12512.919212] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [12517.491974] Key type lgssc unregistered [12518.119286] LNet: 320793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12518.144165] LNetError: 320793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12518.209587] LNet: Removed LNI 192.168.203.101@tcp [12519.777276] Key type .llcrypt unregistered [12519.778869] Key type ._llcrypt unregistered [12559.513072] Key type ._llcrypt registered [12559.516822] Key type .llcrypt registered [12559.648886] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [12561.180117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12561.453335] alg: No test for adler32 (adler32-zlib) [12562.539513] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [12562.796352] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [12564.455092] Key type lgssc registered [12566.002705] Lustre: Echo OBD driver; http://www.lustre.org/ [12576.831070] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12576.852735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12578.808921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12586.897827] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12589.046310] LustreError: 322214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12589.064561] LustreError: 322214:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12594.146056] LustreError: 322215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12599.270635] LustreError: 322214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12599.348891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12600.095508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12607.236396] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12618.657055] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12627.300464] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12639.561557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12640.291555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12649.934964] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12661.813634] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12671.968332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12671.983478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12672.007132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12673.772423] LustreError: 322210:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12674.031474] Lustre: server umount lustre-MDT0000 complete [12674.529877] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12678.653413] LustreError: 322195:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785400951 with bad export cookie 9238285866525606234 [12678.664578] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12678.860460] LustreError: 323437:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12678.885885] LustreError: 323437:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12678.892612] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.1@tcp (stopping) [12679.649338] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12679.665060] Lustre: Skipped 1 previous similar message [12683.681879] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12683.693562] Lustre: Skipped 1 previous similar message [12685.558820] Lustre: server umount lustre-MDT0001 complete [12696.790655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12711.603650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12723.699096] LustreError: 321522:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c62b6850 x1872128465997568/t0(0) o250->MGC192.168.203.101@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12724.489086] LustreError: 324051:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12724.520488] LustreError: 324051:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12724.627061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12724.736525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [12731.235672] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12740.079667] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12741.484581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12742.071018] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12747.248401] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12748.485549] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12760.491821] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12763.540376] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12775.463565] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12778.127209] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [12786.152308] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12786.170730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12786.199907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12786.213303] Lustre: Skipped 1 previous similar message [12788.197597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12791.727747] Lustre: server umount lustre-OST0000 complete [12798.433193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12798.433383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12798.468563] Lustre: Skipped 1 previous similar message [12798.488975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12798.492214] Lustre: Skipped 2 previous similar messages [12803.571087] LustreError: 326553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12803.604190] LustreError: 326553:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [12804.110284] Lustre: server umount lustre-MDT0000 complete [12809.816205] LustreError: 322195:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785401082 with bad export cookie 9238285866525613402 [12809.827314] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12809.842464] LustreError: 322195:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [12810.625181] Lustre: server umount lustre-MDT0001 complete [12833.246953] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [12838.042643] Key type lgssc unregistered [12838.474537] LNet: 328513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12838.489866] LNetError: 328513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12838.502660] LNet: Removed LNI 192.168.203.101@tcp [12841.079247] Key type .llcrypt unregistered [12841.080894] Key type ._llcrypt unregistered [12876.578242] Key type ._llcrypt registered [12876.582159] Key type .llcrypt registered [12876.810147] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [12879.199611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12879.270536] alg: No test for adler32 (adler32-zlib) [12880.738160] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [12881.357577] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [12883.239178] Key type lgssc registered [12885.466693] Lustre: Echo OBD driver; http://www.lustre.org/ [12897.680197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12897.724126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12899.782659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12907.056574] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12910.051601] LustreError: 329933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12910.098895] LustreError: 329933:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12915.181449] LustreError: 329934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12920.292862] LustreError: 329933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12921.140222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12921.822678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12929.881917] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12943.562877] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12952.558974] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12956.646736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12956.650047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12956.667791] Lustre: Skipped 1 previous similar message [12956.703720] Lustre: Skipped 1 previous similar message [12961.761214] LustreError: 331099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12961.837443] Lustre: server umount lustre-MDT0000 complete [12966.796158] LustreError: 329913:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785401239 with bad export cookie 12373112200288942867 [12966.822555] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12967.350553] Lustre: server umount lustre-MDT0001 complete [12986.117166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12986.605617] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12986.665918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12986.961247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12994.801189] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13007.497351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13007.602273] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13007.648252] Lustre: 332526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13007.894432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13013.695536] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13024.895716] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13033.210632] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13046.697147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13046.969276] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [13046.991315] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [13047.008308] LustreError: 334412:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [13047.022028] LustreError: 334412:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [13047.027989] LustreError: 334412:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [13047.140365] Lustre: server umount lustre-OST0000 complete [13047.151530] LustreError: 334412:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [13063.034723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13063.652519] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [13064.125715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13074.256120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [13076.624780] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13088.575495] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13099.058708] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13099.124852] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [13099.179545] Lustre: lustre-OST0001: new disk, initializing [13099.182810] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [13099.191181] Lustre: 335959:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13099.341177] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13106.017828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13106.036946] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13106.288065] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13109.290100] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13122.633507] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13131.745325] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13131.770865] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13131.773900] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [13131.825857] Lustre: Skipped 1 previous similar message [13136.901750] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [13136.904823] Lustre: Skipped 1 previous similar message [13142.072941] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [13142.096597] Lustre: Skipped 1 previous similar message [13144.031920] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13144.172827] Lustre: server umount lustre-OST0001 complete [13147.111078] LustreError: 334879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13147.126202] LustreError: 334879:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13152.259968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13152.263028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13152.284252] Lustre: Skipped 1 previous similar message [13157.352203] LustreError: 334879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13157.384148] LustreError: 334879:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13158.884664] Lustre: server umount lustre-OST0000 complete [13167.074475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13167.084229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13167.109367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13167.117167] Lustre: Skipped 3 previous similar messages [13171.574472] Lustre: server umount lustre-MDT0000 complete [13177.825466] LustreError: 332537:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13177.852262] LustreError: 332537:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13178.258975] LustreError: 332518:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785401450 with bad export cookie 12373112200288943427 [13178.265940] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13178.275657] LustreError: 332518:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 6 previous similar messages [13178.952238] Lustre: server umount lustre-MDT0001 complete [13194.413174] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [13198.990695] Key type lgssc unregistered [13199.886602] LNet: 337922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13199.903743] LNetError: 337922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13199.929766] LNet: Removed LNI 192.168.203.101@tcp [13201.931788] Key type .llcrypt unregistered [13201.936322] Key type ._llcrypt unregistered [13341.020196] Key type ._llcrypt registered [13341.021800] Key type .llcrypt registered [13341.205219] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [13365.981895] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [13368.087356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13368.152051] alg: No test for adler32 (adler32-zlib) [13369.456605] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [13369.985419] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [13371.791147] Key type lgssc registered [13373.508190] Lustre: Echo OBD driver; http://www.lustre.org/ [13419.801468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13419.845125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13421.518043] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13421.571113] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13421.800136] Lustre: lustre-MDT0000: new disk, initializing [13421.972434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13422.043309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13430.367863] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13444.894947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13445.072483] Lustre: 342881:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13445.249372] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13445.261215] Lustre: Skipped 1 previous similar message [13445.467410] Lustre: lustre-MDT0001: new disk, initializing [13445.642888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13445.835870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13445.842902] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13453.574118] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13465.583788] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13473.530775] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13483.186252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13483.440465] Lustre: lustre-OST0000: new disk, initializing [13483.443484] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13483.447389] Lustre: 344849:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13483.574259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13489.226391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13489.248130] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13489.422189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13491.360075] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13502.664951] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13509.121595] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13509.416364] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13514.993360] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13515.371480] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13520.355499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13520.358809] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13520.392192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13520.422047] Lustre: Skipped 1 previous similar message [13524.577812] Lustre: server umount lustre-OST0000 complete [13530.593798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13530.606917] Lustre: Skipped 1 previous similar message [13530.611535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13530.618566] Lustre: Skipped 1 previous similar message [13535.712130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13535.733652] Lustre: Skipped 1 previous similar message [13536.379311] Lustre: server umount lustre-MDT0000 complete [13541.375878] LustreError: 342871:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785401814 with bad export cookie 16967415155069582888 [13541.390839] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13541.393579] LustreError: 342871:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [13542.088459] Lustre: server umount lustre-MDT0001 complete [13559.416532] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 04:57:08 (1785401828) [13575.065771] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 04:57:25 (1785401845) [13585.931285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13586.596951] LustreError: 347610:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13586.623374] LustreError: 347610:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13586.733264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13592.047520] LustreError: 347611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13592.249699] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13597.184696] LustreError: 347610:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13602.283020] LustreError: 347611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13606.718253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13607.176044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13615.760559] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13629.659438] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13637.845854] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13646.495193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13647.103546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13654.957375] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13664.540209] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13673.113715] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [13675.835592] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [13678.578363] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [13680.414686] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [13683.113283] Lustre: Failing over lustre-MDT0000 [13683.190185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13683.213075] Lustre: Skipped 2 previous similar messages [13683.224853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13683.664207] Lustre: server umount lustre-MDT0000 complete [13687.020180] LustreError: 347605:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13697.240416] LustreError: 347607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13697.271442] LustreError: 347607:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [13703.649621] Lustre: 341102:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785401960/real 1785401960] req@0000000067c49547 x1872129312241664/t0(0) o400->MGC192.168.203.101@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785401976 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13703.708892] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13705.896070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13713.894500] LustreError: 347606:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13713.915089] LustreError: 347606:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [13714.915183] LustreError: 341099:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000382a9194 x1872129312246272/t0(0) o250->MGC192.168.203.101@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13715.252301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13715.313309] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [13717.717233] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [13720.568348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [13720.644902] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [13720.796457] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13724.183367] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13739.150147] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13741.610989] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13747.719660] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [13750.765741] Lustre: Failing over lustre-MDT0000 [13751.269544] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13751.288814] Lustre: Skipped 3 previous similar messages [13751.290962] LustreError: 348613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13751.319172] LustreError: 348613:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13751.557741] Lustre: server umount lustre-MDT0000 complete [13766.431203] Lustre: 341101:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785402023/real 1785402023] req@000000004b4fd085 x1872129312266112/t0(0) o400->MGC192.168.203.101@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785402039 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13766.466909] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13774.319835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13776.546332] LustreError: 352003:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [13776.564257] LustreError: 352003:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000616e9436 x1872129312269184/t0(0) o250->MGC192.168.203.101@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1785402049 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13776.601309] LustreError: 352003:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [13778.231734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13778.312533] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [13780.168270] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [13783.553784] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [13783.568525] Lustre: Skipped 2 previous similar messages [13783.602105] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [13783.682416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13785.225689] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13797.586902] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13800.250592] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13813.049283] Lustre: server umount lustre-MDT0000 complete [13814.247509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13814.251561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13814.262475] Lustre: Skipped 1 previous similar message [13814.280399] LustreError: Skipped 1 previous similar message [13817.474801] LustreError: 350385:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785402090 with bad export cookie 16967415155069585597 [13817.478526] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13817.493289] LustreError: 350385:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 1 previous similar message [13818.395146] Lustre: server umount lustre-MDT0001 complete [13833.731670] Lustre: server umount lustre-OST0000 complete [13861.150848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13861.900249] LustreError: 353969:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13861.915876] LustreError: 353969:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 29 previous similar messages [13862.048458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13868.104123] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13881.382990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13881.839985] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [13881.855955] Lustre: Skipped 2 previous similar messages [13889.462926] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13899.341751] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13905.578846] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13915.031805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13925.355977] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [13931.784299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13936.584444] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13947.360455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13947.360505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13947.384978] LustreError: Skipped 1 previous similar message [13947.389051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13947.413284] Lustre: Skipped 2 previous similar messages [13947.448169] Lustre: Skipped 1 previous similar message [13950.514239] Lustre: server umount lustre-OST0000 complete [13963.838210] Lustre: server umount lustre-MDT0000 complete [13970.254297] LustreError: 353951:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785402242 with bad export cookie 16967415155069586346 [13970.268290] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13970.281895] LustreError: 353951:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [13971.128277] Lustre: server umount lustre-MDT0001 complete [13990.207610] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [13994.576265] Key type lgssc unregistered [13995.202508] LNet: 357531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13995.223471] LNetError: 357531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13995.244905] LNet: Removed LNI 192.168.203.101@tcp [13996.760605] Key type .llcrypt unregistered [13996.764743] Key type ._llcrypt unregistered [14032.342776] Key type ._llcrypt registered [14032.347213] Key type .llcrypt registered [14032.557215] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [14034.393312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14034.546063] alg: No test for adler32 (adler32-zlib) [14035.960902] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [14036.609502] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [14038.424051] Key type lgssc registered [14040.768638] Lustre: Echo OBD driver; http://www.lustre.org/ [14055.893820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14056.014527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14057.948523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14065.521532] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [14068.195720] LustreError: 359058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14068.244171] LustreError: 359058:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14073.316689] LustreError: 359059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14077.533325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14077.993631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14085.558193] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [14097.128520] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14107.406466] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14119.574363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14120.349233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14128.587275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [14129.108212] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [14137.312093] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14154.209670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14154.214571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14154.233863] Lustre: Skipped 1 previous similar message [14154.272888] Lustre: Skipped 2 previous similar messages [14158.970055] Lustre: server umount lustre-MDT0000 complete [14159.328990] LustreError: 359788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14159.392215] LustreError: 359788:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14164.452598] LustreError: 359058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14164.488377] LustreError: 359058:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14164.756419] LustreError: 359040:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785402437 with bad export cookie 9179294713750712486 [14164.763086] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14164.779232] LustreError: 359040:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 1 previous similar message [14165.512720] Lustre: server umount lustre-MDT0001 complete [14183.327519] Lustre: server umount lustre-OST0000 complete [14206.439454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14207.367242] LustreError: 362795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14207.490305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14214.719429] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [14217.698186] LustreError: 362796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14217.732684] LustreError: 362796:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14228.781345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14228.971292] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [14229.253360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14235.452806] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [14245.796452] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14253.590960] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14262.403907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14263.058562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14273.984445] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [14278.607735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [14290.973494] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14309.348155] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14309.351890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14309.383179] Lustre: Skipped 2 previous similar messages [14314.495858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14314.510273] Lustre: Skipped 4 previous similar messages [14315.079847] Lustre: server umount lustre-MDT0000 complete [14319.590485] LustreError: 362792:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14319.637163] LustreError: 362792:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [14320.099147] LustreError: 364641:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785402592 with bad export cookie 9179294713750713592 [14320.112128] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14320.118435] LustreError: 364641:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 6 previous similar messages [14321.245721] Lustre: server umount lustre-MDT0001 complete [14337.923932] Lustre: server umount lustre-OST0000 complete [14359.775902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14360.634547] LustreError: 366714:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14360.796141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14367.803617] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [14379.625695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14386.397432] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [14398.411273] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14407.109499] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14417.307299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14417.937658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14417.939755] Lustre: Skipped 1 previous similar message [14426.267406] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [14432.259649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [14436.616715] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14445.536885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14445.546927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14445.572932] Lustre: Skipped 1 previous similar message [14445.607606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14447.589175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14447.615073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14451.093778] Lustre: server umount lustre-OST0000 complete [14462.435325] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14462.464973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14462.480593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14462.485553] Lustre: Skipped 2 previous similar messages [14465.529432] Lustre: server umount lustre-MDT0000 complete [14468.067479] LustreError: 369100:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14468.124967] LustreError: 369100:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [14470.601480] LustreError: 366695:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785402743 with bad export cookie 9179294713750714621 [14470.613139] LustreError: 366695:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 1 previous similar message [14470.627567] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14471.544700] Lustre: server umount lustre-MDT0001 complete [14487.721282] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [14492.232853] Key type lgssc unregistered [14492.706160] LNet: 370251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14492.719039] LNetError: 370251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14492.739322] LNet: Removed LNI 192.168.203.101@tcp [14493.933419] Key type .llcrypt unregistered [14493.936084] Key type ._llcrypt unregistered [14539.746728] Key type ._llcrypt registered [14539.755319] Key type .llcrypt registered [14539.906793] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [14541.682718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14541.916237] alg: No test for adler32 (adler32-zlib) [14543.119623] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [14543.556068] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [14545.311155] Key type lgssc registered [14546.699633] Lustre: Echo OBD driver; http://www.lustre.org/ [14556.769715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14556.803545] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14556.812974] LustreError: 371697:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [14556.840485] LustreError: 371697:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [14556.848197] LustreError: 371697:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [14556.855988] LustreError: 371697:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [14556.866423] LustreError: 371697:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [14566.627743] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [14566.637732] LustreError: 372095:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [14566.662187] LustreError: 372095:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [14566.671671] LustreError: 372095:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [14566.679976] LustreError: 372095:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [14566.689757] LustreError: 372095:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [14601.584772] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [14606.111227] Key type lgssc unregistered [14606.619791] LNet: 373122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14606.631426] LNetError: 373122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14606.647588] LNet: Removed LNI 192.168.203.101@tcp [14608.009553] Key type .llcrypt unregistered [14608.015609] Key type ._llcrypt unregistered [14634.845664] Key type ._llcrypt registered [14634.853134] Key type .llcrypt registered [14635.066480] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [14656.231539] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [14658.776187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14658.815458] alg: No test for adler32 (adler32-zlib) [14660.079779] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [14660.429521] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [14662.167855] Key type lgssc registered [14663.924604] Lustre: Echo OBD driver; http://www.lustre.org/ [14723.719290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14723.746177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14725.261473] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14725.318409] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14725.471979] Lustre: lustre-MDT0000: new disk, initializing [14725.682802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14725.724043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14732.787601] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [14750.827508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14751.110163] Lustre: 376286:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14751.330323] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14751.339921] Lustre: Skipped 1 previous similar message [14751.459182] Lustre: lustre-MDT0001: new disk, initializing [14751.575174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14751.659031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14751.683126] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14759.876255] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [14774.747398] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14784.115151] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14794.114651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14794.312826] Lustre: lustre-OST0000: new disk, initializing [14794.315083] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14794.319024] Lustre: 378258:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14794.443993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14801.576168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14801.601692] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14801.777219] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14804.567504] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [14816.765795] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14825.200051] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14825.526619] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14831.362563] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14831.687108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14837.730455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14837.732251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14837.744695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14837.751561] Lustre: Skipped 1 previous similar message [14841.066054] Lustre: server umount lustre-OST0000 complete [14847.973263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14847.982160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14847.995594] Lustre: Skipped 1 previous similar message [14848.006830] Lustre: Skipped 2 previous similar messages [14853.087610] Lustre: server umount lustre-MDT0000 complete [14858.214300] LustreError: 376295:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14858.239229] LustreError: 376295:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [14858.615877] LustreError: 376279:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785403131 with bad export cookie 15709805889901971927 [14858.623663] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14858.630296] LustreError: 376279:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [14859.742220] Lustre: server umount lustre-MDT0001 complete [14871.938958] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [14874.343212] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 05:19:04 (1785403144) [14883.630099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14884.342063] LustreError: 380722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14884.453597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14888.549743] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [14889.446040] LustreError: 380723:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14894.563215] LustreError: 380722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14899.683928] LustreError: 380723:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14900.284196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14901.128647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14908.558084] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [14922.229915] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14932.111795] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14945.286612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14946.623157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14958.119911] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [14970.203531] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14980.027626] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14980.465060] Lustre: lustre-OST0001: new disk, initializing [14980.482726] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [14980.518985] Lustre: 383567:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14980.748533] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14985.818142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14985.827148] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14985.999120] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14990.519193] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [15003.551036] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15011.812581] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15011.824134] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [15011.825777] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15011.846760] Lustre: Skipped 1 previous similar message [15011.883435] Lustre: Skipped 1 previous similar message [15015.966268] Lustre: server umount lustre-OST0001 complete [15016.942728] LustreError: 382546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15016.969468] LustreError: 382546:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15037.419932] LustreError: 383753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15037.424592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15037.447240] LustreError: 383753:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [15037.451447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15037.451454] Lustre: Skipped 1 previous similar message [15037.516968] Lustre: Skipped 1 previous similar message [15042.528711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15042.530973] Lustre: Skipped 1 previous similar message [15051.231364] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15051.779641] Lustre: server umount lustre-OST0000 complete [15057.810876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15057.822106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15057.845763] Lustre: Skipped 1 previous similar message [15057.862702] Lustre: Skipped 3 previous similar messages [15064.756755] Lustre: server umount lustre-MDT0000 complete [15069.667707] LustreError: 380703:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785403342 with bad export cookie 15709805889901972809 [15069.671846] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15069.677329] LustreError: 380703:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 3 previous similar messages [15076.415816] Lustre: server umount lustre-MDT0001 complete [15097.696768] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [15104.815246] Key type lgssc unregistered [15105.327177] LNet: 385577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15105.372484] LNetError: 385577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15105.405089] LNet: Removed LNI 192.168.203.101@tcp [15108.477682] Key type .llcrypt unregistered [15108.482675] Key type ._llcrypt unregistered [15184.948372] Key type ._llcrypt registered [15184.952903] Key type .llcrypt registered [15185.114794] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [15187.639396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15187.680143] alg: No test for adler32 (adler32-zlib) [15189.179495] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [15189.686819] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [15191.530644] Key type lgssc registered [15193.378486] Lustre: Echo OBD driver; http://www.lustre.org/ [15204.273467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15204.324955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15205.978120] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [15206.019587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15206.287028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15213.327528] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [15224.165953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15224.321096] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [15224.400273] Lustre: 388383:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15224.885274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15232.694500] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [15245.941612] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15258.534616] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15264.231259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15264.262815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15264.288673] Lustre: Skipped 1 previous similar message [15268.732654] Lustre: server umount lustre-MDT0000 complete [15273.243724] LustreError: 389125:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785403545 with bad export cookie 2704359102586650922 [15273.251973] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15273.260939] LustreError: 389125:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [15273.854958] Lustre: server umount lustre-MDT0001 complete [15285.969584] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 05:25:56 (1785403556) [15295.408510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15296.273304] LustreError: 390900:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15296.303816] LustreError: 390900:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15296.383342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15301.611698] LustreError: 390901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15302.694075] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [15306.721584] LustreError: 390900:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15311.844861] LustreError: 390901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15315.037489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15315.610634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15321.892702] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [15332.672203] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15341.240209] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15351.805675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15352.339801] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [15352.937255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15363.385373] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [15376.033421] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15389.878320] Lustre: Permanently deactivating lustre-OST0000 [15389.901436] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [15395.413023] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [15397.573770] LustreError: 393793:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [15397.590213] LustreError: 393793:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [15397.600628] LustreError: 393793:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [15402.505734] LustreError: 393841:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [15402.512362] LustreError: 393841:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [15411.515210] Lustre: server umount lustre-OST0000 complete [15415.801453] LustreError: 394091:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [15415.808273] LustreError: 394091:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [15422.461633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15422.481687] Lustre: Skipped 1 previous similar message [15422.495041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15422.502988] Lustre: Skipped 1 previous similar message [15427.558651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15427.568361] Lustre: Skipped 1 previous similar message [15432.674849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15432.694269] Lustre: Skipped 1 previous similar message [15433.186963] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15434.401630] Lustre: server umount lustre-MDT0000 complete [15437.792829] LustreError: 390900:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15437.854828] LustreError: 390900:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15442.807241] LustreError: 394190:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785403715 with bad export cookie 2704359102586651503 [15442.819959] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15442.845780] LustreError: 394190:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [15453.422524] Lustre: server umount lustre-MDT0001 complete [15464.837737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15471.833163] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [15475.543869] LustreError: 395256:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [15484.085302] Lustre: 395595:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.101@tcp [15486.662107] Lustre: server umount MGS complete [15501.299728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15501.868307] LustreError: 396320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15501.893201] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [15501.909680] Lustre: Skipped 1 previous similar message [15501.989380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15509.254886] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [15522.503467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15522.696981] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [15522.759403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15528.799386] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [15539.248711] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15546.271413] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15549.125932] Lustre: Permanently reactivating lustre-OST0000 [15549.170051] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [15549.172829] Lustre: Skipped 2 previous similar messages [15576.031654] LustreError: 398043:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [15585.474389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15585.839685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15593.544331] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [15607.406388] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15621.604922] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15621.630726] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15621.652860] Lustre: Skipped 1 previous similar message [15621.664071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15624.490934] Lustre: server umount lustre-OST0000 complete [15630.307131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15630.317242] LustreError: Skipped 1 previous similar message [15630.323583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15630.346308] Lustre: Skipped 1 previous similar message [15630.365795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15630.377920] Lustre: Skipped 1 previous similar message [15636.694814] Lustre: server umount lustre-MDT0000 complete [15636.972609] LustreError: 396320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15636.997554] LustreError: 396320:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15641.346844] LustreError: 396300:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785403914 with bad export cookie 2704359102586652546 [15641.357349] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15641.365469] LustreError: 396300:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 1 previous similar message [15642.259134] Lustre: server umount lustre-MDT0001 complete [15663.483303] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [15668.684762] Key type lgssc unregistered [15669.248843] LNet: 400618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15669.256802] LNetError: 400618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15669.300931] LNet: Removed LNI 192.168.203.101@tcp [15671.312800] Key type .llcrypt unregistered [15671.317439] Key type ._llcrypt unregistered [15700.373482] Key type ._llcrypt registered [15700.376595] Key type .llcrypt registered [15700.500702] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [15720.965854] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [15722.231636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15722.254626] alg: No test for adler32 (adler32-zlib) [15723.648495] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [15724.074462] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [15725.815977] Key type lgssc registered [15727.874566] Lustre: Echo OBD driver; http://www.lustre.org/ [15793.086868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15793.121821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15794.547930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15794.616477] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15794.992467] Lustre: lustre-MDT0000: new disk, initializing [15795.203612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15795.255559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15803.472659] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [15817.208945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15817.332864] Lustre: 403782:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15817.378268] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15817.383701] Lustre: Skipped 1 previous similar message [15817.470789] Lustre: lustre-MDT0001: new disk, initializing [15817.566334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15817.607072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15817.615789] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15823.196367] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [15834.811957] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15842.630817] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15851.919769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15852.184427] Lustre: lustre-OST0000: new disk, initializing [15852.190803] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15852.199504] Lustre: 405752:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15852.281039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15860.322565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15860.383310] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15860.693520] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15861.023791] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [15874.435910] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15880.692607] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15880.934477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15887.997938] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15888.276623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15891.427691] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15891.430840] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15891.434532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15891.446294] LustreError: Skipped 1 previous similar message [15891.494341] Lustre: Skipped 1 previous similar message [15896.548900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15896.554909] Lustre: Skipped 1 previous similar message [15897.423940] Lustre: server umount lustre-OST0000 complete [15904.739108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15904.765704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15904.806218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15904.825628] Lustre: Skipped 1 previous similar message [15906.795431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15906.836399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15906.845164] Lustre: Skipped 1 previous similar message [15910.017632] Lustre: server umount lustre-MDT0000 complete [15917.027303] LustreError: 403795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15917.068981] LustreError: 403795:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15917.264616] LustreError: 403775:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785404189 with bad export cookie 10374042791840099193 [15917.301580] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15918.550198] Lustre: server umount lustre-MDT0001 complete [15931.964933] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [15934.118473] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 05:36:44 (1785404204) [15944.036851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15944.491543] LustreError: 408216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15944.623817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15949.793779] LustreError: 408217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15951.466836] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [15954.920802] LustreError: 408216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15960.035545] LustreError: 408217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15963.233489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15963.923274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15970.399707] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [15980.714359] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15989.801669] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15999.849220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16000.446885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16010.633938] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [16021.068394] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16024.163816] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [16027.592904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [16039.392050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16039.399305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16039.425130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16040.423796] LustreError: 410040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16040.640430] Lustre: server umount lustre-OST0000 complete [16050.665130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16050.679342] Lustre: Skipped 1 previous similar message [16050.683582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16050.684596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16050.684600] LustreError: Skipped 1 previous similar message [16050.727278] Lustre: Skipped 1 previous similar message [16053.773098] Lustre: server umount lustre-MDT0000 complete [16058.769546] LustreError: 408196:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785404331 with bad export cookie 10374042791840100075 [16058.786204] LustreError: 408196:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 5 previous similar messages [16058.788687] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16059.416600] Lustre: server umount lustre-MDT0001 complete [16074.553415] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [16078.743930] Key type lgssc unregistered [16079.275532] LNet: 411947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16079.292353] LNetError: 411947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16079.309322] LNet: Removed LNI 192.168.203.101@tcp [16080.540428] Key type .llcrypt unregistered [16080.550388] Key type ._llcrypt unregistered [16112.841084] Key type ._llcrypt registered [16112.845491] Key type .llcrypt registered [16113.038299] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [16113.409306] Key type .llcrypt unregistered [16113.412777] Key type ._llcrypt unregistered [16138.166569] Key type ._llcrypt registered [16138.168720] Key type .llcrypt registered [16138.279566] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [16141.524489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16141.566920] alg: No test for adler32 (adler32-zlib) [16142.855209] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [16143.349727] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [16145.903223] Key type lgssc registered [16148.792183] Lustre: Echo OBD driver; http://www.lustre.org/ [16163.722754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16163.922407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16166.140251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16175.585955] LustreError: 414163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16175.625047] LustreError: 414163:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16176.238237] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [16180.709469] LustreError: 414164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16185.830324] LustreError: 414163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16186.268679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16187.234344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16191.330793] LustreError: 414164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16191.352580] LustreError: 414164:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16191.354713] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [16195.905626] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [16196.576741] LustreError: 414163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16196.611691] LustreError: 414163:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16206.807624] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16206.822202] LustreError: 414164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16206.831488] LustreError: 414164:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16217.474609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16218.374294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16224.542526] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [16237.548978] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16237.566640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16237.598045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16239.095264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16239.095730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16239.107385] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16241.930582] Lustre: server umount lustre-OST0000 complete [16249.319659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16249.322546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16249.340702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16249.368551] Lustre: Skipped 1 previous similar message [16249.395040] Lustre: Skipped 2 previous similar messages [16254.436294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16254.460758] Lustre: Skipped 1 previous similar message [16255.211282] Lustre: server umount lustre-MDT0000 complete [16258.541666] LustreError: 414163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16258.565233] LustreError: 414163:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [16261.574120] LustreError: 414145:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785404534 with bad export cookie 3767806685103689288 [16261.598744] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16261.897594] Lustre: server umount lustre-MDT0001 complete [16275.259881] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [16279.875774] Key type lgssc unregistered [16280.287796] LNet: 417312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16280.312580] LNetError: 417312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16280.353265] LNet: Removed LNI 192.168.203.101@tcp [16282.563614] Key type .llcrypt unregistered [16282.568834] Key type ._llcrypt unregistered [16323.891167] Key type ._llcrypt registered [16323.905912] Key type .llcrypt registered [16324.166466] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [16326.058108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16326.104613] alg: No test for adler32 (adler32-zlib) [16328.275537] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [16328.709549] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [16330.431173] Key type lgssc registered [16332.428109] Lustre: Echo OBD driver; http://www.lustre.org/ [16342.975048] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16343.052319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16358.881395] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [16363.999399] LustreError: 418706:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.101@tcp: failed processing log, type 4: rc = -110 [16390.751248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16402.943800] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [16415.306555] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16425.071234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16425.994220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16426.107882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [16432.109778] LustreError: 419745:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16432.140433] LustreError: 419745:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [16432.839574] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [16437.230645] LustreError: 419744:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16437.289542] LustreError: 419744:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16442.345547] LustreError: 419742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16442.382632] LustreError: 419742:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16444.919325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16445.437805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16452.618175] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [16463.436930] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16471.639654] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16481.253672] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16481.259342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16481.276768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16481.329817] Lustre: Skipped 1 previous similar message [16486.372190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16486.378665] Lustre: Skipped 1 previous similar message [16491.489524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16491.509884] Lustre: Skipped 1 previous similar message [16494.047333] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16494.748252] Lustre: server umount lustre-OST0000 complete [16501.734249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16501.742442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16501.750182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16501.756562] Lustre: Skipped 1 previous similar message [16506.647498] Lustre: server umount lustre-MDT0000 complete [16506.853215] LustreError: 421530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16506.884687] LustreError: 421530:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16511.587716] LustreError: 418712:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785404784 with bad export cookie 13525245346699145821 [16511.602271] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16511.614417] LustreError: 418712:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [16512.413594] Lustre: server umount lustre-MDT0001 complete [16527.792280] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [16533.589442] Key type lgssc unregistered [16534.288242] LNet: 422284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16534.292109] LNetError: 422284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16534.334812] LNet: Removed LNI 192.168.203.101@tcp [16535.977893] Key type .llcrypt unregistered [16535.983275] Key type ._llcrypt unregistered [16570.613189] Key type ._llcrypt registered [16570.617914] Key type .llcrypt registered [16570.760656] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [16574.211703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16574.411448] alg: No test for adler32 (adler32-zlib) [16576.074246] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [16576.716020] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [16578.383185] Key type lgssc registered [16581.035110] Lustre: Echo OBD driver; http://www.lustre.org/ [16592.523298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16592.556953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16594.192709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16599.519660] LustreError: 423705:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16599.559332] LustreError: 423705:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16601.177558] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [16603.627671] LustreError: 423706:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16608.772493] LustreError: 423705:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16612.220852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16612.851817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16620.451869] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [16632.068920] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16642.224225] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16655.085351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16656.247356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16663.434263] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [16665.627229] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [16676.654499] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16684.018203] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16684.029206] LustreError: 423700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16684.073724] Lustre: Skipped 2 previous similar messages [16684.156543] LustreError: 423700:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16684.302369] Lustre: server umount lustre-MDT0000 complete [16688.109582] LustreError: 423685:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.1@tcp arrived at 1785404960 with bad export cookie 9670787236762222454 [16688.130746] LustreError: 423685:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [16691.848685] LustreError: 426289:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785404964 with bad export cookie 9670787236762221558 [16691.876453] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16691.876795] LustreError: 426289:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [16692.192956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16692.196597] LustreError: 423706:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16692.227455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16692.284256] LustreError: 423706:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [16694.896276] Lustre: server umount lustre-OST0000 complete [16701.353391] LustreError: 423706:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16701.396634] LustreError: 423706:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16709.346912] Lustre: server umount lustre-MDT0001 complete [16729.276481] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [16734.068868] Key type lgssc unregistered [16734.532804] LNet: 427416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16734.536368] LNetError: 427416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16734.564232] LNet: Removed LNI 192.168.203.101@tcp [16735.620601] Key type .llcrypt unregistered [16735.626996] Key type ._llcrypt unregistered [16768.612362] Key type ._llcrypt registered [16768.617743] Key type .llcrypt registered [16768.917468] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [16772.292547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16772.383402] alg: No test for adler32 (adler32-zlib) [16773.825601] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [16774.489478] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [16776.391184] Key type lgssc registered [16778.545574] Lustre: Echo OBD driver; http://www.lustre.org/ [16788.131486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16788.158432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16789.675833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16795.133654] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [16800.228956] LustreError: 428836:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16800.266873] LustreError: 428836:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16805.346832] LustreError: 428837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16808.209483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16809.060599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16817.414868] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [16828.122564] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16834.788283] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16842.956627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16843.615354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16851.757532] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [16860.011706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [16861.598938] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16870.193877] LustreError: 430293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [16870.375634] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [16870.388388] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16870.819137] Lustre: server umount lustre-MDT0001 complete [16875.504283] LustreError: 430293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16875.512955] LustreError: 430293:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16880.621101] LustreError: 428832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16880.665841] LustreError: 428832:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16885.501773] Lustre: server umount lustre-OST0000 complete [16890.877947] LustreError: 429725:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16890.941125] LustreError: 429725:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16907.202633] Lustre: server umount lustre-MDT0000 complete [16942.632946] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [16948.916255] Key type lgssc unregistered [16949.437713] LNet: 432552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16949.470195] LNetError: 432552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16949.498124] LNet: Removed LNI 192.168.203.101@tcp [16950.985893] Key type .llcrypt unregistered [16950.992335] Key type ._llcrypt unregistered [16993.768981] Key type ._llcrypt registered [16993.770778] Key type .llcrypt registered [16993.973044] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [17015.969310] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [17017.900607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17018.260560] alg: No test for adler32 (adler32-zlib) [17019.966855] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [17020.603877] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [17022.399180] Key type lgssc registered [17024.366107] Lustre: Echo OBD driver; http://www.lustre.org/ [17081.863471] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [17102.350423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17102.433775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17103.945873] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17104.072777] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17104.251660] Lustre: lustre-MDT0000: new disk, initializing [17104.429934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17104.468569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17111.171822] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [17133.462206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17133.901608] Lustre: lustre-OST0000: new disk, initializing [17133.909327] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17133.915167] Lustre: Skipped 1 previous similar message [17133.930345] Lustre: 437613:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17134.060212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17142.398761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [17142.413114] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [17142.442752] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [17143.401401] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [17156.802466] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17184.122256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17184.471870] Lustre: 436593:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17184.685282] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17184.944417] Lustre: lustre-MDT0001: new disk, initializing [17185.117541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17185.180702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [17185.200724] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [17192.114547] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [17204.101134] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [17204.380898] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [17211.156618] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [17211.580815] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [17216.999959] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17217.004325] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [17217.044717] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [17217.070969] Lustre: Skipped 1 previous similar message [17217.764967] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.1@tcp (stopping) [17222.126624] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [17222.140889] Lustre: Skipped 1 previous similar message [17227.236941] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [17227.267922] Lustre: Skipped 2 previous similar messages [17231.327572] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [17231.923638] Lustre: server umount lustre-MDT0001 complete [17237.494887] LustreError: 436603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17237.550588] LustreError: 436603:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [17238.250396] LustreError: 437620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [17242.593237] LustreError: 436602:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17242.692037] LustreError: 436602:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17247.713684] LustreError: 436603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17247.766333] LustreError: 436603:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17252.852767] LustreError: 437620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17252.904263] LustreError: 437620:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17257.953704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17257.978404] Lustre: Skipped 1 previous similar message [17257.982517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17257.986331] Lustre: Skipped 1 previous similar message [17263.081236] LustreError: 436606:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17263.107273] LustreError: 436606:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17263.871561] Lustre: server umount lustre-OST0000 complete [17275.562789] Lustre: server umount lustre-MDT0000 complete [17310.593254] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [17318.148386] Key type lgssc unregistered [17318.427518] LNet: 441401:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17318.444638] LNetError: 441401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17318.483507] LNet: Removed LNI 192.168.203.101@tcp [17320.050500] Key type .llcrypt unregistered [17320.054282] Key type ._llcrypt unregistered [17376.960417] Key type ._llcrypt registered [17376.967905] Key type .llcrypt registered [17377.096139] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [17377.460321] Key type .llcrypt unregistered [17377.463274] Key type ._llcrypt unregistered [17399.024145] Key type ._llcrypt registered [17399.027682] Key type .llcrypt registered [17399.155950] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [17401.068298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17401.119295] alg: No test for adler32 (adler32-zlib) [17402.548595] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [17403.027450] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [17404.895514] Key type lgssc registered [17406.932111] Lustre: Echo OBD driver; http://www.lustre.org/ [17436.183256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17436.212169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17437.602809] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17437.650298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17437.768491] Lustre: lustre-MDT0000: new disk, initializing [17437.891934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17437.940329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17443.766300] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [17457.674389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17457.877562] Lustre: 444795:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17457.997078] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17458.004910] Lustre: Skipped 1 previous similar message [17458.201171] Lustre: lustre-MDT0001: new disk, initializing [17458.508778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17458.625048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17458.642507] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17464.482840] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [17480.376189] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17491.427466] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17502.063674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17502.741787] Lustre: lustre-OST0000: new disk, initializing [17502.755626] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17502.772237] Lustre: 446765:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17502.970947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17511.840441] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [17512.062415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17512.081597] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17512.225643] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17524.421475] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17534.968102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17534.983598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17535.029864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17538.022146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17538.026629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17540.166411] Lustre: server umount lustre-MDT0000 complete [17544.763489] LustreError: 444786:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785405817 with bad export cookie 15494483762935256137 [17544.767869] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17544.779832] LustreError: 444786:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [17545.787687] Lustre: server umount lustre-MDT0001 complete [17562.525904] Lustre: server umount lustre-OST0000 complete [17574.534572] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 06:04:05 (1785405845) [17585.410087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17586.234251] LustreError: 448794:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17586.258543] LustreError: 448794:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [17586.378629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17591.777366] LustreError: 448795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17594.547385] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [17595.876092] LustreError: 448794:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17600.992523] LustreError: 448795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17606.118501] LustreError: 448794:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17607.091199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17607.784239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17614.777871] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [17625.763679] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17633.688729] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17655.162459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17655.465650] Lustre: Found index 0 for lustre-OST0000, updating log [17655.817896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17656.890880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [17666.261852] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [17678.638930] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17687.524632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17687.534643] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17687.552779] Lustre: Skipped 1 previous similar message [17687.557635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17687.571088] Lustre: Skipped 2 previous similar messages [17692.457977] Lustre: server umount lustre-OST0000 complete [17709.479783] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17709.516981] Lustre: Skipped 1 previous similar message [17709.533915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17709.553765] Lustre: Skipped 3 previous similar messages [17714.657792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17714.669589] Lustre: Skipped 1 previous similar message [17716.082716] Lustre: server umount lustre-MDT0000 complete [17718.564193] LNet: 1 local NIs in recovery (showing 1): 192.168.203.101@tcp [17720.815060] LustreError: 450849:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17720.834419] LustreError: 450849:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17722.534268] LustreError: 448774:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785405995 with bad export cookie 15494483762935257530 [17722.537757] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17722.554598] LustreError: 448774:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [17723.767810] Lustre: server umount lustre-MDT0001 complete [17737.479093] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 06:06:47 (1785406007) [18387.836395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18388.702293] LustreError: 455216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18388.903490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18394.081380] LustreError: 455217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18396.776372] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [18399.206499] LustreError: 455216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18410.341125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18410.465959] LustreError: 455216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18410.499097] LustreError: 455216:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [18411.208214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18417.163751] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [18427.632450] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [18436.677117] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [18448.160075] LNet: 1 local NIs in recovery (showing 1): 192.168.203.101@tcp [18459.513815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18459.856829] Lustre: Found index 0 for lustre-OST0000, updating log [18460.195787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18463.294188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [18467.856791] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [18478.504707] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [18483.681302] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [18483.682082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18483.731464] Lustre: Skipped 1 previous similar message [18483.742297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18483.750621] Lustre: Skipped 2 previous similar messages [18487.533435] Lustre: server umount lustre-OST0000 complete [18496.549150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18496.588658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18496.613541] Lustre: Skipped 1 previous similar message [18498.661255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18498.664012] Lustre: Skipped 2 previous similar messages [18502.094521] Lustre: server umount lustre-MDT0000 complete [18502.893142] LustreError: 455212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18502.922080] LustreError: 455212:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [18509.796582] LustreError: 455196:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785406782 with bad export cookie 15494483762935258615 [18509.812982] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18509.824109] LustreError: 455196:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 3 previous similar messages [18510.642202] LNet: 1 local NIs in recovery (showing 1): 192.168.203.101@tcp [18511.070298] Lustre: server umount lustre-MDT0001 complete [18550.153773] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [18563.134576] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [18621.737253] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 06:21:31 (1785406891) [18646.940615] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [18650.665989] Key type lgssc unregistered [18650.986523] LNet: 461838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18651.002936] LNetError: 461838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18782.136594] LNet: Removed LNI 192.168.203.101@tcp [18783.320875] Key type .llcrypt unregistered [18783.324366] Key type ._llcrypt unregistered [18804.119140] Key type ._llcrypt registered [18804.123117] Key type .llcrypt registered [18804.369279] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [18806.266594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [18806.470406] alg: No test for adler32 (adler32-zlib) [18808.263480] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [18808.785312] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [18810.567177] Key type lgssc registered [18813.256626] Lustre: Echo OBD driver; http://www.lustre.org/ [18821.083199] LNet: Added LNI 192.168.203.101@tcp100 [8/256/0/180] [18822.865847] LNet: Added LNI 192.168.203.101@tcp101 [8/256/0/180] [18824.397297] LNet: Added LNI 192.168.203.101@tcp102 [8/256/0/180] [18825.515779] LNet: Added LNI 192.168.203.101@tcp103 [8/256/0/180] [18827.004973] LNet: Added LNI 192.168.203.101@tcp104 [8/256/0/180] [18828.474747] LNet: Added LNI 192.168.203.101@tcp105 [8/256/0/180] [18829.625407] LNet: Added LNI 192.168.203.101@tcp106 [8/256/0/180] [18831.139035] LNet: Added LNI 192.168.203.101@tcp107 [8/256/0/180] [18832.927554] LNet: Added LNI 192.168.203.101@tcp108 [8/256/0/180] [18834.568390] LNet: Added LNI 192.168.203.101@tcp109 [8/256/0/180] [18835.685352] LNet: Added LNI 192.168.203.101@tcp110 [8/256/0/180] [18836.917136] LNet: Added LNI 192.168.203.101@tcp111 [8/256/0/180] [18838.339727] LNet: Added LNI 192.168.203.101@tcp112 [8/256/0/180] [18839.512656] LNet: Added LNI 192.168.203.101@tcp113 [8/256/0/180] [18840.370171] LNet: Added LNI 192.168.203.101@tcp114 [8/256/0/180] [18841.560625] LNet: Added LNI 192.168.203.101@tcp115 [8/256/0/180] [18842.819119] LNet: Added LNI 192.168.203.101@tcp116 [8/256/0/180] [18843.927673] LNet: Added LNI 192.168.203.101@tcp117 [8/256/0/180] [18844.682558] LNet: Added LNI 192.168.203.101@tcp118 [8/256/0/180] [18845.739887] LNet: Added LNI 192.168.203.101@tcp119 [8/256/0/180] [18846.677349] LNet: Added LNI 192.168.203.101@tcp120 [8/256/0/180] [18848.790043] LNet: Added LNI 192.168.203.101@tcp121 [8/256/0/180] [18850.310616] LNet: Added LNI 192.168.203.101@tcp122 [8/256/0/180] [18852.015532] LNet: Added LNI 192.168.203.101@tcp123 [8/256/0/180] [18853.592907] LNet: Added LNI 192.168.203.101@tcp124 [8/256/0/180] [18855.372416] LNet: Added LNI 192.168.203.101@tcp125 [8/256/0/180] [18857.044390] LNet: Added LNI 192.168.203.101@tcp126 [8/256/0/180] [18858.769344] LNet: Added LNI 192.168.203.101@tcp127 [8/256/0/180] [18860.120210] LNet: Added LNI 192.168.203.101@tcp128 [8/256/0/180] [18861.760692] LNet: Added LNI 192.168.203.101@tcp129 [8/256/0/180] [18863.700558] LNet: Added LNI 192.168.203.101@tcp130 [8/256/0/180] [18864.820757] LNet: Added LNI 192.168.203.101@tcp131 [8/256/0/180] [18866.808834] LNet: Added LNI 192.168.203.101@tcp132 [8/256/0/180] [18868.566664] LNet: Added LNI 192.168.203.101@tcp133 [8/256/0/180] [18869.932451] LNet: Added LNI 192.168.203.101@tcp134 [8/256/0/180] [18871.223047] LNet: Added LNI 192.168.203.101@tcp135 [8/256/0/180] [18883.484465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [18883.548055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18885.160436] Lustre: 464878:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.101@tcp: can't add NID 192.168.203.101@tcp130: rc = -75 [18885.220891] LNetError: 462334:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.101 [18885.262429] LNetError: Protocol error connecting to 192.168.203.101@tcp120 at host 192.168.203.101:988: is it running a compatible version of Lustre? [18885.304673] LNetError: Skipped 3 previous similar messages [18885.647836] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [18885.723336] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [18885.919318] LNet: 2 local NIs in recovery (showing 2): 192.168.203.101@tcp121, 192.168.203.101@tcp120 [18885.995292] Lustre: lustre-MDT0000: new disk, initializing [18886.150383] Lustre: 464936:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.101@tcp: can't add NID 192.168.203.101@tcp130: rc = -75 [18886.177969] Lustre: 464936:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [18886.249083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18886.310750] LNetError: 464899:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.101 [18886.361528] LNetError: 464899:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [18886.367986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [18886.414229] LNetError: Protocol error connecting to 192.168.203.101@tcp104 at host 192.168.203.101:988: is it running a compatible version of Lustre? [18886.445952] LNetError: Skipped 1 previous similar message [18888.992938] LNetError: 464906:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.101 [18889.011727] LNetError: 464906:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [18889.030800] LNetError: Protocol error connecting to 192.168.203.101@tcp120 at host 192.168.203.101:988: is it running a compatible version of Lustre? [18889.062321] LNetError: Skipped 2 previous similar messages [18894.835719] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [18900.256238] LNetError: 464900:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.101 [18900.271185] LNetError: Protocol error connecting to 192.168.203.101@tcp120 at host 192.168.203.101:988: is it running a compatible version of Lustre? [18908.020038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18908.252166] Lustre: 464894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [18908.360649] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [18908.367054] Lustre: Skipped 1 previous similar message [18908.507997] Lustre: lustre-MDT0001: new disk, initializing [18908.571593] Lustre: 465699:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.101@tcp: can't add NID 192.168.203.101@tcp130: rc = -75 [18908.589220] Lustre: 465699:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [18908.780447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18908.830883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [18908.852707] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [18915.543496] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [18916.469442] Lustre: 465983:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.101@tcp: can't add NID 192.168.203.101@tcp130: rc = -75 [18916.485596] Lustre: 465983:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [18916.650387] LNetError: 464904:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.101 [18916.666216] LNetError: Protocol error connecting to 192.168.203.101@tcp120 at host 192.168.203.101:988: is it running a compatible version of Lustre? [18927.587817] LNetError: 464901:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.101 [18927.607506] LNetError: 464901:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [18927.621808] LNetError: Protocol error connecting to 192.168.203.101@tcp118 at host 192.168.203.101:988: is it running a compatible version of Lustre? [18927.643698] LNetError: Skipped 4 previous similar messages [18928.285369] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [18935.450606] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [18943.997467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18944.559562] Lustre: lustre-OST0000: new disk, initializing [18944.570693] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [18944.580173] Lustre: 466872:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [18944.774659] Lustre: 466877:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.101@tcp: can't add NID 192.168.203.101@tcp130: rc = -75 [18944.791886] Lustre: 466877:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [18944.885801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18945.122415] LNetError: 462335:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.101 [18945.149412] LNetError: 462335:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [18945.180818] LNetError: Protocol error connecting to 192.168.203.101@tcp119 at host 192.168.203.101:988: is it running a compatible version of Lustre? [18945.196413] LNetError: Skipped 6 previous similar messages [18945.315411] LNet: 1 local NIs in recovery (showing 1): 192.168.203.101@tcp120 [18953.262855] Lustre: 467183:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.101@tcp: can't add NID 192.168.203.101@tcp130: rc = -75 [18953.273522] Lustre: 467183:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [18953.338167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [18953.360073] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [18953.581542] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [18953.764325] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [18966.532821] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [18979.300646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [18979.304663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18979.314701] LustreError: Skipped 1 previous similar message [18979.319635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18979.335901] Lustre: Skipped 1 previous similar message [18979.369936] Lustre: Skipped 1 previous similar message [18981.988411] LNetError: 464904:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.101 [18982.027609] LNetError: 464904:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 9 previous similar messages [18982.049420] LNetError: Protocol error connecting to 192.168.203.101@tcp116 at host 192.168.203.101:988: is it running a compatible version of Lustre? [18982.088053] LNetError: Skipped 9 previous similar messages [18984.428534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18984.433907] Lustre: Skipped 1 previous similar message [18985.153132] Lustre: server umount lustre-OST0000 complete [18994.657451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18994.683580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18994.697285] Lustre: Skipped 1 previous similar message [18994.738628] Lustre: Skipped 1 previous similar message [18998.070655] Lustre: server umount lustre-MDT0000 complete [19003.842780] LustreError: 464887:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785407276 with bad export cookie 7385483940984020545 [19003.851311] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19003.873677] LustreError: 464887:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [19004.879401] Lustre: server umount lustre-MDT0001 complete [19022.000874] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [19028.779516] Key type lgssc unregistered [19029.588145] LNet: 468789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19029.609341] LNetError: 468789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19029.625758] LNet: Removed LNI 192.168.203.101@tcp [19030.688101] LNet: Removed LNI 192.168.203.101@tcp100 [19030.701957] LNetError: 468789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19030.722121] LNetError: 468789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [19030.764276] LNet: Removed LNI 192.168.203.101@tcp101 [19031.839373] LNet: Removed LNI 192.168.203.101@tcp102 [19031.849157] LNetError: 468789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19031.874212] LNetError: 468789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [19031.920457] LNet: Removed LNI 192.168.203.101@tcp103 [19032.991564] LNet: Removed LNI 192.168.203.101@tcp104 [19034.015152] LNet: Removed LNI 192.168.203.101@tcp105 [19034.017137] LNetError: 468789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19034.040200] LNetError: 468789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [19034.065216] LNet: Removed LNI 192.168.203.101@tcp106 [19035.103202] LNet: Removed LNI 192.168.203.101@tcp107 [19036.128880] LNet: Removed LNI 192.168.203.101@tcp108 [19037.216973] LNet: Removed LNI 192.168.203.101@tcp109 [19038.239124] LNet: Removed LNI 192.168.203.101@tcp110 [19038.300658] LNetError: 468789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19038.326447] LNetError: 468789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [19038.347190] LNet: Removed LNI 192.168.203.101@tcp111 [19039.462317] LNet: Removed LNI 192.168.203.101@tcp112 [19040.544020] LNet: Removed LNI 192.168.203.101@tcp113 [19041.633304] LNet: Removed LNI 192.168.203.101@tcp114 [19042.719121] LNet: Removed LNI 192.168.203.101@tcp115 [19043.743694] LNet: Removed LNI 192.168.203.101@tcp116 [19044.767628] LNet: Removed LNI 192.168.203.101@tcp117 [19045.794269] LNet: Removed LNI 192.168.203.101@tcp118 [19046.816955] LNet: Removed LNI 192.168.203.101@tcp119 [19046.820210] LNetError: 468789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19046.831531] LNetError: 468789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [19046.850227] LNet: Removed LNI 192.168.203.101@tcp120 [19047.904271] LNet: Removed LNI 192.168.203.101@tcp121 [19047.910243] LNet: Removed LNI 192.168.203.101@tcp122 [19047.923183] LNet: Removed LNI 192.168.203.101@tcp123 [19047.933227] LNet: Removed LNI 192.168.203.101@tcp124 [19047.946796] LNet: Removed LNI 192.168.203.101@tcp125 [19047.955443] LNet: Removed LNI 192.168.203.101@tcp126 [19047.963395] LNet: Removed LNI 192.168.203.101@tcp127 [19047.974964] LNet: Removed LNI 192.168.203.101@tcp128 [19047.991336] LNet: Removed LNI 192.168.203.101@tcp129 [19047.993286] LNet: Removed LNI 192.168.203.101@tcp130 [19048.004508] LNet: Removed LNI 192.168.203.101@tcp131 [19048.034835] LNet: Removed LNI 192.168.203.101@tcp132 [19048.047201] LNet: Removed LNI 192.168.203.101@tcp133 [19048.059363] LNet: Removed LNI 192.168.203.101@tcp134 [19048.101601] LNet: Removed LNI 192.168.203.101@tcp135 [19049.280168] Key type .llcrypt unregistered [19049.281602] Key type ._llcrypt unregistered [19079.788429] Key type ._llcrypt registered [19079.791732] Key type .llcrypt registered [19079.970699] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [19105.786371] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [19108.527118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19108.548564] alg: No test for adler32 (adler32-zlib) [19111.229865] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [19112.000794] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [19113.887259] Key type lgssc registered [19116.273707] Lustre: Echo OBD driver; http://www.lustre.org/ [19173.676780] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 06:30:43 (1785407443) [19200.487357] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [19204.291333] Key type lgssc unregistered [19204.666763] LNet: 472890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19204.685633] LNetError: 472890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19204.722327] LNet: Removed LNI 192.168.203.101@tcp [19205.788250] Key type .llcrypt unregistered [19205.791901] Key type ._llcrypt unregistered [19230.242474] Key type ._llcrypt registered [19230.247708] Key type .llcrypt registered [19230.412535] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [19232.086419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19232.404924] alg: No test for adler32 (adler32-zlib) [19233.924786] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [19234.670912] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [19236.738098] Key type lgssc registered [19238.662149] Lustre: Echo OBD driver; http://www.lustre.org/ [19251.413731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [19251.454178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19252.884821] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [19253.013124] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [19253.179040] Lustre: lustre-MDT0000: new disk, initializing [19253.353061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19253.403603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [19259.947406] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [19272.504172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19272.681444] Lustre: 474110:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [19272.724340] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [19272.728075] Lustre: Skipped 1 previous similar message [19272.820636] Lustre: lustre-MDT0001: new disk, initializing [19272.943623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19272.981635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [19272.992938] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [19280.614256] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [19291.004525] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19298.523086] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19310.132642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19310.897105] Lustre: lustre-OST0000: new disk, initializing [19310.904482] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [19310.917624] Lustre: 476078:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [19311.151654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19319.466072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [19319.504877] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [19319.722592] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [19321.279922] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [19334.429471] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [19340.392609] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing lnet_if_list [19348.193707] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [19348.235795] LNet: Added LNI 192.168.203.101@tcp10 [8/256/0/180] [19357.372283] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [19357.431025] LNet: Added LNI 192.168.203.101@tcp11 [8/256/0/180] [19365.534631] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [19365.582457] LNet: Added LNI 192.168.203.101@tcp12 [8/256/0/180] [19372.964432] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [19373.011532] LNet: Added LNI 192.168.203.101@tcp13 [8/256/0/180] [19379.681807] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [19379.792908] LNet: Added LNI 192.168.203.101@tcp14 [8/256/0/180] [19386.285126] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [19386.326990] LNet: Added LNI 192.168.203.101@tcp15 [8/256/0/180] [19393.377077] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [19393.410681] LNet: Added LNI 192.168.203.101@tcp16 [8/256/0/180] [19400.702130] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [19400.780553] LNet: Added LNI 192.168.203.101@tcp17 [8/256/0/180] [19410.543775] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [19410.613971] LNet: Added LNI 192.168.203.101@tcp18 [8/256/0/180] [19422.976184] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [19423.092799] LNet: Added LNI 192.168.203.101@tcp19 [8/256/0/180] [19435.245982] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [19435.362228] LNet: Added LNI 192.168.203.101@tcp20 [8/256/0/180] [19442.768945] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [19442.803943] LNet: Added LNI 192.168.203.101@tcp21 [8/256/0/180] [19450.908240] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [19450.947412] LNet: Added LNI 192.168.203.101@tcp22 [8/256/0/180] [19458.807356] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [19458.867393] LNet: Added LNI 192.168.203.101@tcp23 [8/256/0/180] [19468.775553] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [19468.864476] LNet: Added LNI 192.168.203.101@tcp24 [8/256/0/180] [19480.962891] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [19481.004065] LNet: Added LNI 192.168.203.101@tcp25 [8/256/0/180] [19491.024274] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [19491.123255] LNet: Added LNI 192.168.203.101@tcp26 [8/256/0/180] [19503.158019] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [19503.198871] LNet: Added LNI 192.168.203.101@tcp27 [8/256/0/180] [19511.637731] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [19511.705077] LNet: Added LNI 192.168.203.101@tcp28 [8/256/0/180] [19522.497441] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [19522.571207] LNet: Added LNI 192.168.203.101@tcp29 [8/256/0/180] [19532.748419] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [19532.795631] LNet: Added LNI 192.168.203.101@tcp30 [8/256/0/180] [19543.909527] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [19543.976520] LNet: Added LNI 192.168.203.101@tcp31 [8/256/0/180] [19555.155822] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [19555.205579] LNet: Added LNI 192.168.203.101@tcp32 [8/256/0/180] [19565.967938] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [19566.020712] LNet: Added LNI 192.168.203.101@tcp33 [8/256/0/180] [19573.303515] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [19573.336925] LNet: Added LNI 192.168.203.101@tcp34 [8/256/0/180] [19581.787897] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [19581.806780] LNet: Added LNI 192.168.203.101@tcp35 [8/256/0/180] [19589.670084] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [19589.739636] LNet: Added LNI 192.168.203.101@tcp36 [8/256/0/180] [19604.802441] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [19604.867646] LNet: Added LNI 192.168.203.101@tcp37 [8/256/0/180] [19620.519713] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [19620.578110] LNet: Added LNI 192.168.203.101@tcp38 [8/256/0/180] [19630.015119] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [19630.059373] LNet: Added LNI 192.168.203.101@tcp39 [8/256/0/180] [19639.907846] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [19639.953077] LNet: Added LNI 192.168.203.101@tcp40 [8/256/0/180] [19649.534387] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [19649.607223] LNet: Added LNI 192.168.203.101@tcp41 [8/256/0/180] [19663.767551] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [19663.790266] LNet: Added LNI 192.168.203.101@tcp42 [8/256/0/180] [19681.583345] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [19681.691109] LNet: Added LNI 192.168.203.101@tcp43 [8/256/0/180] [19692.226629] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [19692.257812] LNet: Added LNI 192.168.203.101@tcp44 [8/256/0/180] [19704.768119] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [19704.843423] LNet: Added LNI 192.168.203.101@tcp45 [8/256/0/180] [19727.376390] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [19727.393704] LNet: Added LNI 192.168.203.101@tcp46 [8/256/0/180] [19742.339235] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [19742.424703] LNet: Added LNI 192.168.203.101@tcp47 [8/256/0/180] [19753.067722] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [19753.090346] LNet: Added LNI 192.168.203.101@tcp48 [8/256/0/180] [19764.129946] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [19764.243710] LNet: Added LNI 192.168.203.101@tcp49 [8/256/0/180] [19776.834319] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [19776.859339] LNet: Added LNI 192.168.203.101@tcp50 [8/256/0/180] [19805.667407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [19805.673801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19805.691646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19806.180838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19806.210174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19806.218515] Lustre: Skipped 2 previous similar messages [19811.275525] Lustre: server umount lustre-MDT0000 complete [19816.431525] LustreError: 475096:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19816.450893] LustreError: 475096:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [19817.667169] LustreError: 474102:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785408090 with bad export cookie 16155226834887080840 [19817.681067] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19818.195395] Lustre: server umount lustre-MDT0001 complete [19837.919923] Lustre: 473382:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785408094/real 1785408094] req@00000000873cdd64 x1872135462288896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785408110 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19837.960206] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19837.987288] Lustre: Skipped 1 previous similar message [19838.947472] Lustre: 473381:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785408095/real 1785408095] req@00000000fd84d11f x1872135462289024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785408111 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19843.040120] Lustre: 473380:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785408099/real 1785408099] req@000000006bd4070d x1872135462289152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785408115 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19849.188628] Lustre: 473382:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785408105/real 1785408105] req@00000000cb82be32 x1872135462289536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785408121 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19849.241272] Lustre: 473382:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [19850.544961] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [19854.306864] Lustre: 473381:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785408110/real 1785408110] req@000000004105f399 x1872135462289664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785408126 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19865.659565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19888.444289] LustreError: 476071:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19888.508048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19893.495127] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [19893.753120] LustreError: 476072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19896.811388] LustreError: 476071:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19896.845014] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [19901.925195] LustreError: 476072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19903.911583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19904.622804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19907.701404] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [19910.779195] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [19922.829078] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19932.626926] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19955.168626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [19955.175986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19955.195900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19959.278220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19959.285211] Lustre: Skipped 2 previous similar messages [19961.676485] Lustre: server umount lustre-OST0000 complete [19969.510308] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19969.512956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19969.524960] Lustre: Skipped 2 previous similar messages [19969.540687] Lustre: Skipped 1 previous similar message [19975.766561] Lustre: server umount lustre-MDT0000 complete [19979.744991] LustreError: 501442:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19979.795782] LustreError: 501442:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [19981.948462] LustreError: 474101:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785408254 with bad export cookie 16155226834887083542 [19981.958180] LustreError: 474101:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [19981.966653] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19982.428282] Lustre: server umount lustre-MDT0001 complete [20001.463738] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [20009.036178] Key type lgssc unregistered [20010.148694] LNet: 503178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [20010.183992] LNetError: 503178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [20011.233463] LNet: Removed LNI 192.168.203.101@tcp [20011.235252] LNetError: 503178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [20011.258509] LNet: Removed LNI 192.168.203.101@tcp10 [20011.278475] LNet: Removed LNI 192.168.203.101@tcp11 [20011.287325] LNet: Removed LNI 192.168.203.101@tcp12 [20011.325087] LNet: Removed LNI 192.168.203.101@tcp13 [20011.329124] LNet: Removed LNI 192.168.203.101@tcp14 [20011.339091] LNet: Removed LNI 192.168.203.101@tcp15 [20011.343411] LNet: Removed LNI 192.168.203.101@tcp16 [20011.357751] LNet: Removed LNI 192.168.203.101@tcp17 [20011.385055] LNet: Removed LNI 192.168.203.101@tcp18 [20011.394693] LNet: Removed LNI 192.168.203.101@tcp19 [20011.423995] LNet: Removed LNI 192.168.203.101@tcp20 [20011.439242] LNet: Removed LNI 192.168.203.101@tcp21 [20011.448956] LNet: Removed LNI 192.168.203.101@tcp22 [20011.477868] LNet: Removed LNI 192.168.203.101@tcp23 [20011.479515] LNet: Removed LNI 192.168.203.101@tcp24 [20011.492321] LNet: Removed LNI 192.168.203.101@tcp25 [20011.504916] LNet: Removed LNI 192.168.203.101@tcp26 [20011.511961] LNet: Removed LNI 192.168.203.101@tcp27 [20011.523686] LNet: Removed LNI 192.168.203.101@tcp28 [20011.528993] LNet: Removed LNI 192.168.203.101@tcp29 [20011.560930] LNet: Removed LNI 192.168.203.101@tcp30 [20011.583936] LNet: Removed LNI 192.168.203.101@tcp31 [20011.611705] LNet: Removed LNI 192.168.203.101@tcp32 [20011.613337] LNet: Removed LNI 192.168.203.101@tcp33 [20011.627589] LNet: Removed LNI 192.168.203.101@tcp34 [20011.638408] LNet: Removed LNI 192.168.203.101@tcp35 [20011.648979] LNet: Removed LNI 192.168.203.101@tcp36 [20011.667159] LNet: Removed LNI 192.168.203.101@tcp37 [20011.684023] LNet: Removed LNI 192.168.203.101@tcp38 [20011.685640] LNet: Removed LNI 192.168.203.101@tcp39 [20011.695706] LNet: Removed LNI 192.168.203.101@tcp40 [20011.697457] LNet: Removed LNI 192.168.203.101@tcp41 [20011.710702] LNet: Removed LNI 192.168.203.101@tcp42 [20011.721645] LNet: Removed LNI 192.168.203.101@tcp43 [20011.732015] LNet: Removed LNI 192.168.203.101@tcp44 [20011.736927] LNet: Removed LNI 192.168.203.101@tcp45 [20011.743610] LNet: Removed LNI 192.168.203.101@tcp46 [20011.745297] LNet: Removed LNI 192.168.203.101@tcp47 [20011.753351] LNet: Removed LNI 192.168.203.101@tcp48 [20011.761979] LNet: Removed LNI 192.168.203.101@tcp49 [20011.789977] LNet: Removed LNI 192.168.203.101@tcp50 [20013.784079] Key type .llcrypt unregistered [20013.791900] Key type ._llcrypt unregistered [20049.561535] Key type ._llcrypt registered [20049.567645] Key type .llcrypt registered [20049.696135] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [20074.209593] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [20075.568806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [20075.693832] alg: No test for adler32 (adler32-zlib) [20077.090848] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [20077.776363] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [20079.567119] Key type lgssc registered [20082.794083] Lustre: Echo OBD driver; http://www.lustre.org/ [20146.105487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [20146.149248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20147.534900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [20147.606207] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [20147.839481] Lustre: lustre-MDT0000: new disk, initializing [20148.044132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20148.082677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [20157.391872] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [20174.405724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20174.602680] Lustre: 506385:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [20174.710747] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [20174.719151] Lustre: Skipped 1 previous similar message [20174.871548] Lustre: lustre-MDT0001: new disk, initializing [20175.068437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [20175.186211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [20175.224846] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [20181.505668] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [20199.689902] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [20213.750315] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [20224.758171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20225.002484] Lustre: *** cfs_fail_loc=90f, val=0*** [20225.180523] Lustre: lustre-OST0000: new disk, initializing [20225.188650] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [20225.208976] Lustre: 508406:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [20225.423805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [20232.864984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [20232.884497] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [20233.060137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [20234.250469] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [20245.498127] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [20263.926643] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [20263.928271] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20263.949266] LustreError: Skipped 1 previous similar message [20263.965452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20263.978915] Lustre: Skipped 1 previous similar message [20266.470391] Lustre: server umount lustre-OST0000 complete [20272.608798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [20272.614622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20272.642778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20272.647710] Lustre: Skipped 1 previous similar message [20274.145281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20274.145958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20274.183639] Lustre: Skipped 1 previous similar message [20278.310797] Lustre: server umount lustre-MDT0000 complete [20285.044744] LustreError: 506378:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785408557 with bad export cookie 4009958942027820392 [20285.048769] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20285.069090] LustreError: 506378:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [20286.063750] Lustre: server umount lustre-MDT0001 complete [20302.664745] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [20308.357841] Key type lgssc unregistered [20309.155223] LNet: 510315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [20309.175316] LNetError: 510315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [20309.238788] LNet: Removed LNI 192.168.203.101@tcp [20311.683354] Key type .llcrypt unregistered [20311.685287] Key type ._llcrypt unregistered [20345.634274] Key type ._llcrypt registered [20345.640739] Key type .llcrypt registered [20345.805318] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [20348.372288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [20348.505339] alg: No test for adler32 (adler32-zlib) [20349.696374] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [20349.958785] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [20351.687204] Key type lgssc registered [20353.230806] Lustre: Echo OBD driver; http://www.lustre.org/ [20366.769386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [20366.836729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20368.983887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20379.109538] LustreError: 511592:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20379.236880] LustreError: 511592:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [20379.906377] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [20384.244955] LustreError: 511591:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20389.349342] LustreError: 511592:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20393.452633] LustreError: 511591:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20397.983150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20398.440252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [20406.878720] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [20417.777831] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [20426.745241] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [20436.560329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20437.102123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [20445.504053] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [20454.421695] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [20468.535237] loop0: detected capacity change from 0 to 204800000 [20469.097480] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20469.115129] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20469.168381] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20476.401210] loop0: detected capacity change from 0 to 204800000 [20476.735163] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20476.752330] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20476.796154] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20483.593246] loop0: detected capacity change from 0 to 204800000 [20490.796043] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20491.144281] Lustre: fs15246-OST0000: new disk, initializing [20491.157648] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [20491.176745] Lustre: 515189:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [20491.297844] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [20501.139283] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [20520.062717] loop1: detected capacity change from 0 to 204800000 [20530.193135] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20530.382234] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [20530.487704] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [20530.693256] Lustre: fs15246-MDT0000: new disk, initializing [20531.143346] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20531.245262] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [20531.454410] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [20532.514471] Lustre: 516314:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000f3285e37 x1872136631278976/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1785408821 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [20538.376382] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [20541.623662] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [20541.644268] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [20541.691412] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [20574.781272] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20574.827233] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [20576.214295] Lustre: server umount fs15246-MDT0000 complete [20579.824168] LustreError: 511587:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20590.061100] LustreError: 511586:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20590.090510] LustreError: 511586:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [20594.105121] Lustre: server umount fs15246-OST0000 complete [20616.184231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20616.214753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20616.233598] Lustre: Skipped 1 previous similar message [20621.283697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20621.295636] Lustre: Skipped 1 previous similar message [20622.286981] Lustre: server umount lustre-OST0000 complete [20631.522310] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20631.577419] Lustre: Skipped 1 previous similar message [20631.611107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20631.630963] Lustre: Skipped 1 previous similar message [20635.689476] Lustre: server umount lustre-MDT0000 complete [20636.643388] LustreError: 512477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20636.673489] LustreError: 512477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [20642.463925] LustreError: 511571:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785408915 with bad export cookie 10247485956707311680 [20642.490539] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20643.740991] Lustre: server umount lustre-MDT0001 complete [20661.995309] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [20668.179284] Key type lgssc unregistered [20669.026597] LNet: 519436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [20669.041423] LNetError: 519436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [20670.053789] LNet: Removed LNI 192.168.203.101@tcp [20672.335915] Key type .llcrypt unregistered [20672.338079] Key type ._llcrypt unregistered [20709.599454] Key type ._llcrypt registered [20709.616804] Key type .llcrypt registered [20709.842948] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [20711.769332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [20711.837229] alg: No test for adler32 (adler32-zlib) [20713.161601] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [20713.653727] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [20715.375155] Key type lgssc registered [20717.232736] Lustre: Echo OBD driver; http://www.lustre.org/ [20769.933061] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [20785.576392] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [20838.118258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [20838.163754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20839.617788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [20839.675333] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [20839.819282] Lustre: lustre-MDT0000: new disk, initializing [20840.034800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20840.109375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [20847.412186] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [20867.078474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20867.413937] Lustre: 523929:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [20867.514149] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [20867.522502] Lustre: Skipped 1 previous similar message [20867.759220] Lustre: lustre-MDT0001: new disk, initializing [20868.047642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [20868.185908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [20868.205550] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [20875.030897] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [20889.707567] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [20896.981688] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [20906.221488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20906.608370] Lustre: lustre-OST0000: new disk, initializing [20906.616413] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [20906.626776] Lustre: 525901:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [20906.773050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [20912.281502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [20912.294214] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [20912.496067] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [20915.348197] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [20927.028557] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [20932.975206] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [20933.398572] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [20938.707677] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [20939.019331] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [20943.328321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [20943.331491] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20943.350494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20943.362141] Lustre: Skipped 1 previous similar message [20943.387046] Lustre: Skipped 1 previous similar message [20948.841858] Lustre: server umount lustre-OST0000 complete [20955.123668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [20955.139282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20955.181179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20958.693477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20958.693920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20958.729991] Lustre: Skipped 1 previous similar message [20960.478169] Lustre: server umount lustre-MDT0000 complete [20963.809689] LustreError: 523937:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20963.842309] LustreError: 523937:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [20965.240523] LustreError: 523921:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785409237 with bad export cookie 14854237902755477455 [20965.256433] LustreError: 523921:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 1 previous similar message [20965.271126] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20965.926460] Lustre: server umount lustre-MDT0001 complete [20998.127443] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [21017.737565] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [21059.113289] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 20595 sec ========= 07:02:09 (1785409329) [21061.822166] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:02:12 (1785409332) === [21066.871604] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:02:17 (1785409337) === [21127.701395] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [21133.415648] Key type lgssc unregistered [21133.958066] LNet: 532604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [21133.968244] LNetError: 532604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [21134.001635] LNet: Removed LNI 192.168.203.101@tcp [21135.584892] Key type .llcrypt unregistered [21135.587597] Key type ._llcrypt unregistered