[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 473561493 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002463] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008016] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009007] pid_max: default: 32768 minimum: 301 [ 0.010134] LSM: Security Framework initializing [ 0.011038] Yama: becoming mindful. [ 0.012029] SELinux: Initializing. [ 0.014020] *** VALIDATE selinux *** [ 0.022525] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026329] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027116] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029083] *** VALIDATE tmpfs *** [ 0.031349] *** VALIDATE proc *** [ 0.032193] *** VALIDATE cgroup *** [ 0.033006] *** VALIDATE cgroup2 *** [ 0.034239] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035125] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037022] Spectre V2 : User space: Vulnerable [ 0.038003] Speculative Store Bypass: Vulnerable [ 0.041093] debug: unmapping init [mem 0xffffffffa9c59000-0xffffffffa9c60fff] [ 0.043870] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044562] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045012] ... version: 2 [ 0.046007] ... bit width: 48 [ 0.047007] ... generic registers: 4 [ 0.048007] ... value mask: 0000ffffffffffff [ 0.049006] ... max period: 00007fffffffffff [ 0.050007] ... fixed-purpose events: 3 [ 0.051007] ... event mask: 000000070000000f [ 0.052210] rcu: Hierarchical SRCU implementation. [ 0.054286] smp: Bringing up secondary CPUs ... [ 0.055386] x86: Booting SMP configuration: [ 0.056016] .... node #0, CPUs: #1 #2 #3 [ 0.059012] smp: Brought up 1 node, 4 CPUs [ 0.061007] smpboot: Max logical packages: 1 [ 0.062009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.146651] node 0 deferred pages initialised in 83ms [ 0.148557] devtmpfs: initialized [ 0.149181] x86/mm: Memory block size: 128MB [ 0.151565] gcov: version magic: 0x41383552 [ 0.152558] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.153072] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154252] pinctrl core: initialized pinctrl subsystem [ 0.155128] [ 0.155552] ************************************************************* [ 0.156006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157005] ** ** [ 0.158005] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.159006] ** ** [ 0.160005] ** This means that this kernel is built to expose internal ** [ 0.161006] ** IOMMU data structures, which may compromise security on ** [ 0.162006] ** your system. ** [ 0.163005] ** ** [ 0.164006] ** If you see this message and you are not debugging the ** [ 0.165006] ** kernel, report this immediately to your vendor! ** [ 0.166007] ** ** [ 0.167007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168005] ************************************************************* [ 0.169569] NET: Registered protocol family 16 [ 0.170387] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.171033] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.172035] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.173353] cpuidle: using governor menu [ 0.175197] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.177451] PCI: Using configuration type 1 for base access [ 0.178139] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.186073] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.187027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.189027] cryptd: max_cpu_qlen set to 1000 [ 0.191898] ACPI: Added _OSI(Module Device) [ 0.193011] ACPI: Added _OSI(Processor Device) [ 0.195014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.196010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.201282] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.207429] ACPI: Interpreter enabled [ 0.209053] ACPI: PM: (supports S0 S3 S4 S5) [ 0.210010] ACPI: Using IOAPIC for interrupt routing [ 0.211099] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.214301] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.222866] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.225040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.227007] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.229042] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.234269] acpiphp: Slot [2] registered [ 0.235120] acpiphp: Slot [5] registered [ 0.237118] acpiphp: Slot [6] registered [ 0.239108] acpiphp: Slot [7] registered [ 0.241075] acpiphp: Slot [8] registered [ 0.242062] acpiphp: Slot [9] registered [ 0.243063] acpiphp: Slot [10] registered [ 0.244046] acpiphp: Slot [3] registered [ 0.245063] acpiphp: Slot [4] registered [ 0.247079] acpiphp: Slot [11] registered [ 0.248063] acpiphp: Slot [12] registered [ 0.249053] acpiphp: Slot [13] registered [ 0.250064] acpiphp: Slot [14] registered [ 0.250973] acpiphp: Slot [15] registered [ 0.252049] acpiphp: Slot [16] registered [ 0.254044] acpiphp: Slot [17] registered [ 0.255054] acpiphp: Slot [18] registered [ 0.256060] acpiphp: Slot [19] registered [ 0.258133] acpiphp: Slot [20] registered [ 0.260096] acpiphp: Slot [21] registered [ 0.261153] acpiphp: Slot [22] registered [ 0.263074] acpiphp: Slot [23] registered [ 0.264119] acpiphp: Slot [24] registered [ 0.265080] acpiphp: Slot [25] registered [ 0.267069] acpiphp: Slot [26] registered [ 0.268064] acpiphp: Slot [27] registered [ 0.269059] acpiphp: Slot [28] registered [ 0.271083] acpiphp: Slot [29] registered [ 0.272060] acpiphp: Slot [30] registered [ 0.274092] acpiphp: Slot [31] registered [ 0.275076] PCI host bridge to bus 0000:00 [ 0.276012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.278011] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.280015] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.283014] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.286014] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.289014] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.290177] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.292976] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.296248] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.307734] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.312564] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.315011] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.317011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.319021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.321556] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.323716] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.325026] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.327529] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.331009] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.341015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.344007] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.347083] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.352007] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.355000] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.367011] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.376548] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.379976] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.386010] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.398015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.405420] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.408910] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.420016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.433017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.442652] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.451013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.456944] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.472015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.479736] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.484011] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.487938] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.501011] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.509932] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.515007] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.519010] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.534013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.541124] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.545394] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.547294] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.549230] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.551166] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.556268] iommu: Default domain type: Passthrough [ 0.558453] SCSI subsystem initialized [ 0.559076] ACPI: bus type USB registered [ 0.560059] usbcore: registered new interface driver usbfs [ 0.561041] usbcore: registered new interface driver hub [ 0.562049] usbcore: registered new device driver usb [ 0.564134] pps_core: LinuxPPS API ver. 1 registered [ 0.565005] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.567031] PTP clock support registered [ 0.569059] EDAC MC: Ver: 3.0.0 [ 0.570139] PCI: Using ACPI for IRQ routing [ 0.571563] NetLabel: Initializing [ 0.572005] NetLabel: domain hash size = 128 [ 0.572842] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.574049] NetLabel: unlabeled traffic allowed by default [ 0.576160] vgaarb: loaded [ 0.577284] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.580011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.588075] clocksource: Switched to clocksource kvm-clock [ 0.691302] VFS: Disk quotas dquot_6.6.0 [ 0.692160] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.693634] *** VALIDATE ramfs *** [ 0.694332] *** VALIDATE hugetlbfs *** [ 0.695897] pnp: PnP ACPI init [ 0.698239] pnp: PnP ACPI: found 6 devices [ 0.720306] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.722659] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.724214] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.725766] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.727289] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.729638] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.731612] NET: Registered protocol family 2 [ 0.733365] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.736592] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.740706] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.746073] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.749252] TCP: Hash tables configured (established 65536 bind 65536) [ 0.752720] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.755662] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.758349] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.761328] NET: Registered protocol family 1 [ 0.763630] RPC: Registered named UNIX socket transport module. [ 0.766026] RPC: Registered udp transport module. [ 0.768035] RPC: Registered tcp transport module. [ 0.769858] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.772187] NET: Registered protocol family 44 [ 0.773258] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.775124] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.776470] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.778378] PCI: CLS 0 bytes, default 64 [ 0.780152] Unpacking initramfs... [ 2.019537] debug: unmapping init [mem 0xffff8f24bcc54000-0xffff8f24bffbffff] [ 2.021840] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.023232] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.025027] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.494019] Initialise system trusted keyrings [ 2.495441] Key type blacklist registered [ 2.497829] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.505656] zbud: loaded [ 2.508140] *** VALIDATE nfs *** [ 2.509377] *** VALIDATE nfs4 *** [ 2.510965] pstore: using deflate compression [ 2.514136] Platform Keyring initialized [ 2.610668] NET: Registered protocol family 38 [ 2.614984] Key type asymmetric registered [ 2.616544] Asymmetric key parser 'x509' registered [ 2.618515] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.621750] io scheduler mq-deadline registered [ 2.623330] io scheduler kyber registered [ 2.625288] io scheduler bfq registered [ 2.627368] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.630646] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.633430] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.636732] ACPI: Power Button [PWRF] [ 2.642146] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.649189] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.660754] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.666912] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.688653] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.714989] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.741545] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.745695] Non-volatile memory driver v1.3 [ 2.746901] Linux agpgart interface v0.103 [ 2.775404] virtio_blk virtio1: [vda] 150008 512-byte logical blocks (76.8 MB/73.2 MiB) [ 2.777784] vda: detected capacity change from 0 to 76804096 [ 2.788642] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.790936] vdb: detected capacity change from 0 to 1073741824 [ 2.803529] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.806034] vdc: detected capacity change from 0 to 2621440000 [ 2.817393] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.819642] vdd: detected capacity change from 0 to 2621440000 [ 2.831444] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.834422] vde: detected capacity change from 0 to 4294967296 [ 2.846961] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.849263] vdf: detected capacity change from 0 to 4294967296 [ 2.854714] libphy: Fixed MDIO Bus: probed [ 2.859249] usbcore: registered new interface driver usbserial_generic [ 2.862268] usbserial: USB Serial support registered for generic [ 2.864368] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.867774] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.869209] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.871059] mousedev: PS/2 mouse device common for all mice [ 2.874623] rtc_cmos 00:05: RTC can wake from S4 [ 2.876099] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.879337] rtc_cmos 00:05: registered as rtc0 [ 2.881264] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.884918] intel_pstate: CPU model not supported [ 2.887824] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.888348] hid: raw HID events driver (C) Jiri Kosina [ 2.895929] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.896076] usbcore: registered new interface driver usbhid [ 2.902888] usbhid: USB HID core driver [ 2.904340] drop_monitor: Initializing network drop monitor service [ 2.906975] Initializing XFRM netlink socket [ 2.909113] NET: Registered protocol family 10 [ 2.912407] Segment Routing with IPv6 [ 2.913769] NET: Registered protocol family 17 [ 2.915739] mpls_gso: MPLS GSO support [ 2.919609] RAS: Correctable Errors collector initialized. [ 2.920647] AVX version of gcm_enc/dec engaged. [ 2.921485] AES CTR mode by8 optimization enabled [ 3.001389] sched_clock: Marking stable (3001371247, 0)->(3839180754, -837809507) [ 3.004912] registered taskstats version 1 [ 3.006687] Loading compiled-in X.509 certificates [ 3.008405] zswap: loaded using pool lzo/zbud [ 3.029802] Key type big_key registered [ 3.041947] Key type encrypted registered [ 3.043395] ima: No TPM chip found, activating TPM-bypass! [ 3.045333] ima: Allocated hash algorithm: sha1 [ 3.046881] ima: No architecture policies found [ 3.048656] evm: Initialising EVM extended attributes: [ 3.050384] evm: security.selinux [ 3.051654] evm: security.ima [ 3.052663] evm: security.capability [ 3.053922] evm: HMAC attrs: 0x1 [ 3.055994] rtc_cmos 00:05: setting system clock to 2026-09-15 04:57:14 UTC (1789448234) [ 3.062362] debug: unmapping init [mem 0xffffffffaac03000-0xffffffffaadfffff] [ 3.065503] debug: unmapping init [mem 0xffffffffa9982000-0xffffffffa9c58fff] [ 3.075111] Write protecting the kernel read-only data: 28672k [ 3.078146] debug: unmapping init [mem 0xffffffffa8003000-0xffffffffa81fffff] [ 3.081232] debug: unmapping init [mem 0xffffffffa8914000-0xffffffffa89fffff] [ 3.114341] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.122986] systemd[1]: Detected virtualization kvm. [ 3.124467] systemd[1]: Detected architecture x86-64. [ 3.126023] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.151712] systemd[1]: No hostname configured. [ 3.153430] systemd[1]: Set hostname to . [ 3.155406] random: systemd: uninitialized urandom read (16 bytes read) [ 3.157965] systemd[1]: Initializing machine ID from random generator. [ 3.191626] random: ln: uninitialized urandom read (6 bytes read) [ 3.268671] random: systemd: uninitialized urandom read (16 bytes read) [ 3.270886] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.278452] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.282241] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.931948] device-mapper: uevent: version 1.0.3 [ 3.934585] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.604498] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook...[ 4.620430] random: fast init done [ 4.844448] scsi host0: ata_piix [ 4.865226] scsi host1: ata_piix [ 4.866738] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.868760] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.496026] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.559745] random: crng init done [ 9.561253] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.077225] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.400284] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.862601] SELinux: Disabled at runtime. [ 11.950214] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.962982] systemd[1]: Detected virtualization kvm. [ 11.966860] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.693070] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.698686] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.705908] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.716511] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.721690] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.733621] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.754593] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... Startin[ 12.837858] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS g Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.739726] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.290468] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.354296] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.598324] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.621385] EDAC sbridge: Ver: 1.1.2 [ 17.289464] Key type dns_resolver registered [ 17.690150] NFS: Registering the id_resolver key type [ 17.694168] Key type id_resolver registered [ 17.695451] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg434-server login: [ 49.060170] libcfs: loading out-of-tree module taints kernel. [ 49.082137] Key type ._llcrypt registered [ 49.083531] Key type .llcrypt registered [ 49.137582] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 57.771538] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 58.452770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 58.461343] alg: No test for adler32 (adler32-zlib) [ 59.505198] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 59.884790] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 61.527284] Key type lgssc registered [ 62.217453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 69.003915] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 82.577262] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 87.189108] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 87.202846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 88.293913] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 88.304685] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 88.346897] Lustre: lustre-MDT0000: new disk, initializing [ 88.370675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 88.377115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 89.751818] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 95.072190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 95.109342] Lustre: 6489:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 95.121077] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 95.123159] Lustre: Skipped 1 previous similar message [ 95.158881] Lustre: lustre-MDT0001: new disk, initializing [ 95.183517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 95.193534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 95.197922] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 96.670042] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 99.034956] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 102.307554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 102.399649] Lustre: lustre-OST0000: new disk, initializing [ 102.401886] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 102.404956] Lustre: 8427:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 102.423753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 103.979863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 103.983237] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 104.002677] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 104.376505] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 109.863615] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 109.905249] Lustre: lustre-OST0001: new disk, initializing [ 109.906403] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 109.908373] Lustre: 9501:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 109.923355] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 111.945540] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 113.835360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 113.839371] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 113.851776] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 117.789467] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 124.401549] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 126.282640] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing check_logdir /tmp/testlogs/ [ 127.825941] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing yml_node [ 129.266912] Lustre: DEBUG MARKER: Client: 2.17.58.40 [ 130.154574] Lustre: DEBUG MARKER: MDS: 2.17.58.40 [ 131.062425] Lustre: DEBUG MARKER: OSS: 2.17.58.40 [ 131.654586] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 15 00:59:21 EDT 2026 [ 137.362349] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 137.847504] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 149.983710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 149.984075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 149.986933] Lustre: Skipped 3 previous similar messages [ 149.988479] Lustre: Skipped 2 previous similar messages [ 155.104023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 155.106894] Lustre: Skipped 3 previous similar messages [ 155.543414] Lustre: server umount lustre-MDT0000 complete [ 156.745549] LustreError: 6481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789448388 with bad export cookie 7901009758215734500 [ 156.747205] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 156.749227] LustreError: 6481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 160.223673] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 160.224055] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 160.228477] Lustre: Skipped 1 previous similar message [ 162.914676] Lustre: server umount lustre-MDT0001 complete [ 170.432485] Lustre: server umount lustre-OST0000 complete [ 186.335112] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 186.387847] Lustre: server umount lustre-OST0001 complete [ 188.550962] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 191.703240] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 204.065113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 204.180079] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 204.194513] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 204.229195] Lustre: lustre-MDT0000: new disk, initializing [ 204.250382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.259293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 205.724579] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 210.145918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 210.197845] Lustre: 15038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 210.307659] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 211.785831] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 215.613564] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 217.829926] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 220.693469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 220.852242] Lustre: lustre-OST0000: new disk, initializing [ 220.854838] Lustre: Skipped 1 previous similar message [ 220.857997] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 220.860966] Lustre: Skipped 2 previous similar messages [ 220.864981] Lustre: 17008:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 220.911057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 220.914276] Lustre: Skipped 1 previous similar message [ 222.735969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 222.744033] Lustre: Skipped 1 previous similar message [ 222.750158] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 222.794547] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 223.764366] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 229.472107] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 232.746276] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 232.918937] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 235.382862] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 235.525169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 238.049667] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 238.055531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 238.056712] LustreError: Skipped 1 previous similar message [ 238.058251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 238.058257] Lustre: Skipped 3 previous similar messages [ 238.080150] Lustre: Skipped 1 previous similar message [ 242.818699] Lustre: server umount lustre-OST0000 complete [ 246.243270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 246.252296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 246.261023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 246.272229] Lustre: Skipped 1 previous similar message [ 251.581236] Lustre: server umount lustre-MDT0000 complete [ 253.408758] LustreError: 15045:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 253.437308] LustreError: 15045:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 256.613133] LustreError: 15834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789448488 with bad export cookie 7901009758215736019 [ 256.622947] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 256.642841] LustreError: 15834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 257.264340] Lustre: server umount lustre-MDT0001 complete [ 259.922436] hrtimer: interrupt took 6663834 ns [ 266.127926] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 01:01:33 (1789448493) [ 301.477971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 302.329743] LustreError: 19897:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 302.479339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 307.691787] LustreError: 19898:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 310.961944] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 311.783791] LustreError: 19897:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.896762] LustreError: 19898:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 327.140986] LustreError: 19897:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 327.175639] LustreError: 19897:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 328.933775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 338.412216] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 350.959677] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 358.854766] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 361.954915] Lustre: Failing over lustre-MDT0000 [ 362.322528] Lustre: server umount lustre-MDT0000 complete [ 365.024557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 365.047140] Lustre: Skipped 1 previous similar message [ 365.053671] LustreError: 19897:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 365.077554] LustreError: 19897:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 381.284067] Lustre: 3627:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789448596/real 1789448596] req@ffff8f252c96a300 x1876372531310208/t0(0) o400->MGC192.168.204.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789448612 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 381.356666] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 383.454892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 390.633589] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6da6068385581059 [ 390.640544] Lustre: MGC192.168.204.134@tcp: Connection restored to 0@lo (at 0@lo) [ 390.984355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 390.988166] Lustre: Skipped 1 previous similar message [ 396.270365] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 396.279086] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 396.311974] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 396.334334] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 406.894938] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 411.616417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 411.621495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 411.625389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 411.625396] Lustre: Skipped 2 previous similar messages [ 411.669031] Lustre: Skipped 2 previous similar messages [ 417.526801] Lustre: server umount lustre-MDT0000 complete [ 421.621646] LustreError: 19879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789448653 with bad export cookie 7901009758215737433 [ 421.640375] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 422.060826] Lustre: server umount lustre-MDT0001 complete [ 432.164697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 432.827593] LustreError: 23243:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 432.849057] LustreError: 23243:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 438.475890] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 448.197827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 453.581699] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 462.029457] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 468.431243] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 470.958780] Lustre: Failing over lustre-MDT0000 [ 471.208865] Lustre: server umount lustre-MDT0000 complete [ 473.064945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 473.096628] Lustre: Skipped 1 previous similar message [ 489.312547] Lustre: 3625:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789448704/real 1789448704] req@ffff8f24029be680 x1876372531339264/t0(0) o400->MGC192.168.204.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789448720 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 489.370132] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 491.670358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 498.594301] LustreError: 23244:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 498.638381] LustreError: 23244:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 17 previous similar messages [ 498.904423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 498.906716] Lustre: Skipped 2 previous similar messages [ 503.192426] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 504.293127] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 504.299383] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 504.302899] Lustre: Skipped 1 previous similar message [ 504.313071] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 512.954631] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 516.731755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.733565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 516.741169] Lustre: Skipped 1 previous similar message [ 516.753253] Lustre: Skipped 4 previous similar messages [ 523.028824] Lustre: server umount lustre-MDT0000 complete [ 526.955915] LustreError: 23225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789448758 with bad export cookie 7901009758215738371 [ 526.971522] LustreError: 23225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 526.977836] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 527.302133] Lustre: server umount lustre-MDT0001 complete [ 536.496635] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 01:06:05 (1789448765) [ 553.518333] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 565.404448] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 609.914125] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 622.261832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 622.458506] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 622.482577] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 622.552981] Lustre: lustre-MDT0000: new disk, initializing [ 622.709207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 627.733627] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 639.189884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 639.330580] Lustre: 29614:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 639.585604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 639.593402] Lustre: Skipped 1 previous similar message [ 639.652349] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 643.699136] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 653.234779] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 658.674586] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 665.647625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 666.091796] Lustre: lustre-OST0000: new disk, initializing [ 666.098728] Lustre: Skipped 1 previous similar message [ 666.102938] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 666.109858] Lustre: Skipped 2 previous similar messages [ 666.124633] Lustre: 31584:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 667.684908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 667.693842] Lustre: Skipped 1 previous similar message [ 667.702703] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 667.760691] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 667.792906] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 667.804447] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 673.946988] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 685.223690] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 691.364670] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 714.215183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 714.220930] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 714.230033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 714.230043] Lustre: Skipped 2 previous similar messages [ 714.281679] Lustre: Skipped 1 previous similar message [ 725.983708] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 726.123201] Lustre: server umount lustre-OST0000 complete [ 731.620626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 731.629316] LustreError: Skipped 1 previous similar message [ 736.454810] Lustre: server umount lustre-MDT0000 complete [ 739.810118] LustreError: 29625:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 739.843849] LustreError: 29625:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 741.257288] LustreError: 30418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789448972 with bad export cookie 7901009758215738777 [ 741.272239] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 741.278849] LustreError: 30418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 741.954902] Lustre: server umount lustre-MDT0001 complete [ 755.450684] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 758.724391] Key type lgssc unregistered [ 759.085500] LNet: 33564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.094563] LNetError: 33564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 759.102981] LNet: Removed LNI 192.168.204.134@tcp [ 760.199165] Key type .llcrypt unregistered [ 760.200756] Key type ._llcrypt unregistered [ 788.023737] Key type ._llcrypt registered [ 788.029033] Key type .llcrypt registered [ 788.239750] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 789.463076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 789.564873] alg: No test for adler32 (adler32-zlib) [ 790.860904] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 791.168107] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 792.935204] Key type lgssc registered [ 794.651641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 803.914305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 803.944894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 805.664436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 811.682535] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 816.114837] LustreError: 34974:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 816.134782] LustreError: 34974:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 821.232707] LustreError: 34975:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 821.979900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 822.427907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 828.367104] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 835.725696] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 841.770911] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 848.797405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 849.358177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 851.443547] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 857.046300] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 866.125658] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 866.740729] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 882.637510] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 01:11:51 (1789449111) [ 884.582272] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 886.333585] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 893.085143] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 01:12:01 (1789449121) [ 903.642907] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 01:12:12 (1789449132) [ 904.797119] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 914.809570] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 914.816671] Lustre: Skipped 2 previous similar messages [ 917.140283] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 01:12:25 (1789449145) [ 919.062931] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 919.070285] Lustre: Skipped 2 previous similar messages [ 927.977667] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 927.984118] Lustre: Skipped 4 previous similar messages [ 941.397722] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 01:12:50 (1789449170) [ 944.299734] Lustre: *** cfs_fail_loc=131b, val=2*** [ 948.257706] Lustre: *** cfs_fail_loc=131b, val=2*** [ 958.500065] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 01:13:07 (1789449187) [ 960.650087] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 960.654915] Lustre: Skipped 6 previous similar messages [ 975.311859] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 01:13:24 (1789449204) [ 994.799147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 994.815985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 994.826216] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 994.828147] Lustre: Skipped 1 previous similar message [ 994.859469] Lustre: Skipped 1 previous similar message [ 998.256387] Lustre: server umount lustre-OST0000 complete [ 1005.037650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1005.042517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1005.057746] Lustre: Skipped 1 previous similar message [ 1005.073885] Lustre: Skipped 1 previous similar message [ 1008.891666] Lustre: server umount lustre-MDT0000 complete [ 1010.152238] LustreError: 39813:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1010.176545] LustreError: 39813:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1014.130752] LustreError: 35708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789449245 with bad export cookie 11983550175942035253 [ 1014.144175] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1014.708805] Lustre: server umount lustre-MDT0001 complete [ 1028.367667] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 1031.062593] Key type lgssc unregistered [ 1031.411593] LNet: 42837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1031.419821] LNetError: 42837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1032.495640] LNet: Removed LNI 192.168.204.134@tcp [ 1033.881610] Key type .llcrypt unregistered [ 1033.889394] Key type ._llcrypt unregistered [ 1059.132816] Key type ._llcrypt registered [ 1059.139223] Key type .llcrypt registered [ 1059.399727] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 1077.173839] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 1078.841432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1078.941706] alg: No test for adler32 (adler32-zlib) [ 1080.105978] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 1080.368889] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 1082.040142] Key type lgssc registered [ 1083.464803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1122.911352] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1122.943731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1124.276180] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1124.316646] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1124.442732] Lustre: lustre-MDT0000: new disk, initializing [ 1124.534500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1124.556808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1130.570825] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1142.858062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1142.965967] Lustre: 46003:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1143.049286] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1143.055383] Lustre: Skipped 1 previous similar message [ 1143.193531] Lustre: lustre-MDT0001: new disk, initializing [ 1143.293569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1143.324946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1143.336847] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1148.473270] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1159.380293] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1165.887120] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1173.250358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1173.509690] Lustre: lustre-OST0000: new disk, initializing [ 1173.515416] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1173.526501] Lustre: 47975:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1173.674561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1178.740621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1178.760242] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1178.950994] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1181.298572] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1192.850667] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1198.928739] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1199.202780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1204.743791] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1205.030517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1209.831692] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1209.841050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1209.844500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1209.857069] Lustre: Skipped 1 previous similar message [ 1213.704907] Lustre: server umount lustre-OST0000 complete [ 1220.068946] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1220.070479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1220.070702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1220.070707] LustreError: Skipped 1 previous similar message [ 1220.084769] Lustre: Skipped 1 previous similar message [ 1220.089835] Lustre: Skipped 1 previous similar message [ 1223.677921] Lustre: server umount lustre-MDT0000 complete [ 1228.258650] LustreError: 45996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789449459 with bad export cookie 14101932001545871523 [ 1228.262083] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1228.268806] LustreError: 45996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1228.672959] Lustre: server umount lustre-MDT0001 complete [ 1239.105156] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 01:17:48 (1789449468) [ 1245.850356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1246.446380] LustreError: 50435:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1246.461861] LustreError: 50435:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1246.513604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1251.466796] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1251.813221] LustreError: 50436:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1255.976767] LustreError: 50435:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1260.501840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1260.982222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1265.986874] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1273.660463] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1279.598971] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1287.159864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1287.652896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1295.257853] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1303.190909] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1311.220572] LustreError: 52964:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1312.370650] Lustre: Modifying parameter general.timeout=1 in log params [ 1313.470155] Lustre: Modifying parameter general.timeout=2 in log params [ 1315.654708] Lustre: Modifying parameter general.timeout=4 in log params [ 1315.659050] Lustre: Skipped 1 previous similar message [ 1319.695590] Lustre: Modifying parameter general.timeout=8 in log params [ 1319.699265] Lustre: Skipped 3 previous similar messages [ 1327.724871] Lustre: Modifying parameter general.timeout=16 in log params [ 1327.729786] Lustre: Skipped 7 previous similar messages [ 1343.825256] Lustre: Modifying parameter general.timeout=31 in log params [ 1343.832531] Lustre: Skipped 14 previous similar messages [ 1376.039437] Lustre: Modifying parameter general.timeout=64 in log params [ 1376.049994] Lustre: Skipped 32 previous similar messages [ 1445.205552] Lustre: Disabling parameter general.timeout= in log params [ 1445.208755] Lustre: Skipped 65 previous similar messages [ 1448.444551] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 01:21:17 (1789449677) [ 1451.742468] Lustre: 59917:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1453.732732] Lustre: 60015:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1455.907480] Lustre: 60127:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1460.038214] Lustre: 60327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1460.052542] Lustre: 60327:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1492.849533] Lustre: 61800:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1492.860176] Lustre: 61800:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1492.895594] LustreError: 61800:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1497.471112] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 01:22:06 (1789449726) [ 1511.907303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1511.921418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1511.935955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1516.526372] Lustre: server umount lustre-MDT0000 complete [ 1517.547564] LustreError: 51163:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1517.572072] LustreError: 51163:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1520.408262] LustreError: 50415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789449751 with bad export cookie 14101932001545872405 [ 1520.417758] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1520.426370] LustreError: 50415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1520.831551] Lustre: server umount lustre-MDT0001 complete [ 1533.801249] Lustre: server umount lustre-OST0000 complete [ 1562.540833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1562.973343] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1563.471785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1568.834509] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1578.470911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1578.641167] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1578.671045] Lustre: 64330:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1583.150793] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1592.021493] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1598.553453] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1606.010173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1606.278671] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1606.586789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1606.595632] Lustre: Skipped 1 previous similar message [ 1614.764950] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1623.521800] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1625.909995] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1625.924205] Lustre: Skipped 23 previous similar messages [ 1626.003946] Lustre: 66897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1637.344676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1637.357628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1637.377576] Lustre: Skipped 2 previous similar messages [ 1637.391647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1637.402679] Lustre: Skipped 3 previous similar messages [ 1640.066465] Lustre: server umount lustre-OST0000 complete [ 1645.537224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1645.551327] LustreError: Skipped 1 previous similar message [ 1645.563799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1645.577158] Lustre: Skipped 1 previous similar message [ 1645.586403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1645.593282] Lustre: Skipped 1 previous similar message [ 1658.336632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1658.351178] Lustre: Skipped 5 previous similar messages [ 1659.359144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1659.776242] Lustre: server umount lustre-MDT0000 complete [ 1663.469291] LustreError: 64341:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1663.491714] LustreError: 64341:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1664.397756] LustreError: 64321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789449895 with bad export cookie 14101932001545875513 [ 1664.406365] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1664.409202] LustreError: 64321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1664.965592] Lustre: server umount lustre-MDT0001 complete [ 1678.069874] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 1681.810640] Key type lgssc unregistered [ 1682.159748] LNet: 68106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1682.164304] LNetError: 68106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1682.179292] LNet: Removed LNI 192.168.204.134@tcp [ 1683.252346] Key type .llcrypt unregistered [ 1683.254970] Key type ._llcrypt unregistered [ 1711.250614] Key type ._llcrypt registered [ 1711.253853] Key type .llcrypt registered [ 1711.359775] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 1713.109558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1713.148049] alg: No test for adler32 (adler32-zlib) [ 1714.350619] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 1714.605774] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 1716.320266] Key type lgssc registered [ 1717.677642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1726.744854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1726.806265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1728.643955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1733.809344] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1738.210827] LustreError: 69520:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1738.238408] LustreError: 69520:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1743.329536] LustreError: 69519:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1743.710227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1744.410690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1749.352446] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1757.649518] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1763.693269] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1770.818045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1771.604672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1778.166484] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1786.860362] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1797.615083] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1809.892523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1809.900554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1809.912712] Lustre: Skipped 2 previous similar messages [ 1809.933975] Lustre: Skipped 2 previous similar messages [ 1812.873053] Lustre: server umount lustre-MDT0000 complete [ 1815.012071] LustreError: 69519:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1815.027732] LustreError: 69519:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1816.993106] LustreError: 69500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789450048 with bad export cookie 1951869610781090667 [ 1816.996371] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1817.011628] LustreError: 69500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1817.386160] Lustre: server umount lustre-MDT0001 complete [ 1832.238471] Lustre: server umount lustre-OST0000 complete [ 1862.179884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1862.491964] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1862.525986] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1862.747082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1867.324458] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1876.732827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1876.820062] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1876.860898] Lustre: 74391:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1877.420858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1882.383700] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1890.108330] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1896.103191] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1902.500221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1902.759100] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1903.214531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1909.449089] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1917.352565] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1919.173177] Lustre: 76970:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1921.294911] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1929.185504] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1929.195495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1929.221273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1929.697114] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1932.875724] Lustre: server umount lustre-OST0000 complete [ 1938.401646] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1938.413672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1938.429447] Lustre: Skipped 1 previous similar message [ 1938.433427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1938.437335] Lustre: Skipped 1 previous similar message [ 1942.917697] Lustre: server umount lustre-MDT0000 complete [ 1945.064971] LustreError: 74403:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1945.089638] LustreError: 74403:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1946.657160] LustreError: 76972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789450178 with bad export cookie 1951869610781091493 [ 1946.667034] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1946.668847] LustreError: 76972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1947.138337] Lustre: server umount lustre-MDT0001 complete [ 1958.970453] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 1961.794146] Key type lgssc unregistered [ 1962.086372] LNet: 78181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1962.099469] LNetError: 78181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1962.112827] LNet: Removed LNI 192.168.204.134@tcp [ 1962.974194] Key type .llcrypt unregistered [ 1962.975784] Key type ._llcrypt unregistered [ 1987.534934] Key type ._llcrypt registered [ 1987.537438] Key type .llcrypt registered [ 1987.696629] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 1988.798478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1988.905313] alg: No test for adler32 (adler32-zlib) [ 1990.439845] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 1990.872183] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 1992.656458] Key type lgssc registered [ 1993.983374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2002.671487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2002.688639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2004.171957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2008.924908] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2013.671381] LustreError: 79590:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2013.703526] LustreError: 79590:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2016.918083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2017.700742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2022.403288] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2029.663899] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2035.141711] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2042.281209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2042.822205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2049.669514] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2057.477632] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2067.228700] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2068.122291] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2069.164263] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2071.405554] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2071.409457] Lustre: Skipped 1 previous similar message [ 2075.603253] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2075.607330] Lustre: Skipped 3 previous similar messages [ 2084.239882] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2084.247652] Lustre: Skipped 7 previous similar messages [ 2100.607704] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2100.614696] Lustre: Skipped 17 previous similar messages [ 2132.649255] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2132.653825] Lustre: Skipped 29 previous similar messages [ 2178.507451] Lustre: 87697:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2181.553370] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 01:33:30 (1789450410) [ 2183.318031] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2185.119992] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 01:33:34 (1789450414) [ 2190.310100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2190.311542] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2190.313156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2190.358956] Lustre: Skipped 1 previous similar message [ 2195.472538] Lustre: server umount lustre-OST0000 complete [ 2200.544380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2200.560932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2200.562410] Lustre: Skipped 1 previous similar message [ 2200.571592] Lustre: Skipped 2 previous similar messages [ 2205.671445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2205.687393] Lustre: Skipped 1 previous similar message [ 2205.941415] Lustre: server umount lustre-MDT0000 complete [ 2209.761918] LustreError: 79585:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2209.789093] LustreError: 79585:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2210.017886] LustreError: 79569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789450441 with bad export cookie 121277599439312365 [ 2210.020135] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2210.029808] LustreError: 79569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2210.479107] Lustre: server umount lustre-MDT0001 complete [ 2223.489979] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 2226.989689] Key type lgssc unregistered [ 2227.477743] LNet: 88945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2227.487452] LNetError: 88945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2227.522025] LNet: Removed LNI 192.168.204.134@tcp [ 2228.739672] Key type .llcrypt unregistered [ 2228.741984] Key type ._llcrypt unregistered [ 2234.700386] Key type ._llcrypt registered [ 2234.703400] Key type .llcrypt registered [ 2234.830304] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2240.791904] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules [ 2242.072567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2242.213430] alg: No test for adler32 (adler32-zlib) [ 2243.440702] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 2243.706279] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 2245.391253] Key type lgssc registered [ 2246.818627] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2248.223403] LustreError: 90144:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2266.550576] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 2270.249531] LustreError: 90144:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2276.584819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2276.599522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2278.075849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2283.182628] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2287.585388] LustreError: 91140:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2287.620733] LustreError: 91140:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2292.710100] LustreError: 91139:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2293.064724] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 01:35:21 (1789450521) [ 2297.829717] LustreError: 91140:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2298.788328] LustreError: 91171:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2298.808212] LustreError: 91171:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2298.824448] LustreError: 91171:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2299.309582] Lustre: server umount lustre-MDT0000 complete [ 2315.436801] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 2318.442816] Key type lgssc unregistered [ 2318.834865] LNet: 92666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2318.847446] LNetError: 92666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2318.873901] LNet: Removed LNI 192.168.204.134@tcp [ 2319.939795] Key type .llcrypt unregistered [ 2319.948673] Key type ._llcrypt unregistered [ 2340.858601] Key type ._llcrypt registered [ 2340.862550] Key type .llcrypt registered [ 2341.041460] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 2341.983899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2342.148838] alg: No test for adler32 (adler32-zlib) [ 2343.337353] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 2343.717385] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 2345.423204] Key type lgssc registered [ 2346.773585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2355.385966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2355.412663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2357.028921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2363.068729] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2367.463893] LustreError: 93880:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2367.488665] LustreError: 93880:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2371.734576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2372.490606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2377.083953] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2384.776508] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2390.015832] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2395.832345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2396.562097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2402.158366] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2409.706736] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2458.325418] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 01:38:07 (1789450687) [ 2460.400939] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2462.422512] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 01:38:11 (1789450691) [ 2468.322904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2468.324559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2468.359036] Lustre: Skipped 2 previous similar messages [ 2468.370209] Lustre: Skipped 2 previous similar messages [ 2473.440311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2473.449579] Lustre: Skipped 2 previous similar messages [ 2473.911471] Lustre: server umount lustre-MDT0000 complete [ 2478.272294] LustreError: 93861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789450709 with bad export cookie 8500165714499815583 [ 2478.283183] LustreError: 93861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2478.285661] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2478.568117] LustreError: 93876:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2478.576801] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2479.027632] Lustre: server umount lustre-MDT0001 complete [ 2485.648091] Lustre: server umount lustre-OST0000 complete [ 2497.076942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2497.594522] LustreError: 98165:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2497.611623] LustreError: 98165:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2497.766427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2502.509290] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2503.142225] LustreError: 98166:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.269920] LustreError: 98165:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.028970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2512.847953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2517.804695] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2524.955874] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2530.525979] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2542.616764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2542.826246] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2542.831859] LustreError: 98153:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2542.852768] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2542.861761] LustreError: 100248:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2542.866955] LustreError: 100248:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2542.871151] LustreError: 100248:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2542.941596] Lustre: server umount lustre-OST0000 complete [ 2542.943624] LustreError: 100248:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2551.454147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2551.648113] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2551.660729] LustreError: 98153:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2551.685057] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2551.712186] LustreError: 100713:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2551.724512] LustreError: 100713:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2551.730734] LustreError: 100713:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2551.813163] Lustre: server umount lustre-OST0000 complete [ 2551.816757] LustreError: 100713:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2561.771265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2561.993272] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2562.001492] Lustre: Found index 0 for lustre-OST0000, updating log [ 2562.009933] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2562.099215] Lustre: lustre-OST0000: new disk, initializing [ 2562.102272] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2562.109401] Lustre: 101248:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2562.388834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2568.831210] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2572.504759] Lustre: Failing over lustre-OST0000 [ 2572.805594] Lustre: server umount lustre-OST0000 complete [ 2578.913939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2578.923508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2578.936837] Lustre: Skipped 1 previous similar message [ 2581.987582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2581.997246] Lustre: Skipped 1 previous similar message [ 2587.109438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2587.119119] Lustre: Skipped 1 previous similar message [ 2587.189703] Lustre: server umount lustre-MDT0000 complete [ 2590.735217] LustreError: 98145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789450822 with bad export cookie 8500165714499818320 [ 2590.740324] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2590.742470] LustreError: 98145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2591.084325] Lustre: server umount lustre-MDT0001 complete [ 2600.033748] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 01:40:29 (1789450829) [ 2614.941981] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 2624.997434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2625.520902] LustreError: 103592:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2625.764107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2631.726822] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2643.669849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2651.221652] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2654.593486] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2662.096518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2662.735404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2662.752459] Lustre: Skipped 1 previous similar message [ 2662.767574] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2668.848901] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2668.870883] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2668.882133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2669.589746] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2670.893462] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:07 [ 2670.913832] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2670.946390] Lustre: lustre-OST0000: Recovery over after 0:02, of 1 clients 1 recovered and 0 were evicted. [ 2670.947497] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2670.958993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2678.636974] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2678.746544] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2678.841893] Lustre: lustre-OST0001: new disk, initializing [ 2678.845419] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2678.851160] Lustre: 105914:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2685.387800] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2688.615627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2688.627720] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2688.705518] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2692.784191] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2699.341517] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2704.363476] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2704.365822] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2704.374287] Lustre: Skipped 2 previous similar messages [ 2704.390266] Lustre: Skipped 2 previous similar messages [ 2709.842465] Lustre: server umount lustre-MDT0001 complete [ 2713.479530] LustreError: 103588:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2713.517942] LustreError: 103588:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2718.959461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2719.096064] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2719.101508] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2719.148340] Lustre: 103579:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2720.377473] LustreError: 107454:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2720.385985] LustreError: 107454:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2720.392282] LustreError: 103584:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2720.494335] LustreError: 107463:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.134@tcp: cfg command failed: rc = -17 [ 2720.502442] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2720.502442] [ 2723.767951] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2725.358720] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2734.114512] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 01:42:42 (1789450962) [ 2754.133695] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 2773.583989] Lustre: 109528:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2795.617946] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2799.741632] Lustre: 110663:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2883.553917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2883.559493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2883.577065] Lustre: Skipped 1 previous similar message [ 2883.589996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2883.592729] Lustre: Skipped 4 previous similar messages [ 2886.618789] Lustre: server umount lustre-MDT0000 complete [ 2889.186487] LustreError: 105109:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2889.210044] LustreError: 105109:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 2889.632076] LustreError: 103573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789451121 with bad export cookie 8500165714499819328 [ 2889.634176] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2889.643117] LustreError: 103573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2889.827765] Lustre: server umount lustre-MDT0001 complete [ 2903.620936] Lustre: server umount lustre-OST0000 complete [ 2918.713381] Lustre: server umount lustre-OST0001 complete [ 2925.016307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2936.251858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2946.107426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2957.212577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2977.398536] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 2993.599167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2993.637177] Lustre: lustre-MDT0000: reset Object Index mappings [ 2994.490990] LustreError: 114613:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2994.734223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2994.740029] Lustre: Skipped 2 previous similar messages [ 3001.141509] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3013.423620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3013.519988] Lustre: lustre-MDT0001: reset Object Index mappings [ 3019.970165] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3024.353840] Lustre: 115763:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3032.050683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3039.045872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3039.057404] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3039.280125] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3047.284623] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3053.052420] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3053.059441] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3055.570498] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3063.374832] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3067.156424] Lustre: 117642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3119.266831] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 01:49:07 (1789451347) [ 3145.192948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3145.196573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3145.226457] Lustre: Skipped 6 previous similar messages [ 3145.240018] Lustre: Skipped 7 previous similar messages [ 3150.313601] LustreError: 116951:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3150.330169] LustreError: 116951:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 3150.482164] Lustre: server umount lustre-MDT0000 complete [ 3154.372913] LustreError: 117665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789451385 with bad export cookie 8500165714499866403 [ 3154.380825] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3154.391468] LustreError: 117665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3161.212941] Lustre: server umount lustre-MDT0001 complete [ 3172.463259] Lustre: server umount lustre-OST0000 complete [ 3182.397703] Lustre: server umount lustre-OST0001 complete [ 3189.712401] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 3200.543952] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 3237.171769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3237.465952] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3237.506288] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3237.585148] Lustre: lustre-MDT0000: new disk, initializing [ 3237.675341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3237.681690] Lustre: Skipped 3 previous similar messages [ 3237.695279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3243.024412] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3250.162513] Lustre: server umount lustre-MDT0000 complete [ 3259.795834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3260.180647] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3260.203606] Lustre: 121912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3260.232073] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3264.542849] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3270.311953] Lustre: server umount lustre-MDT0000 complete [ 3278.643931] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 01:51:47 (1789451507) [ 3280.135559] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3282.181264] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 01:51:51 (1789451511) [ 3300.792304] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 3310.858434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3315.623292] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3324.624287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3324.756351] Lustre: 123961:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3324.809765] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3324.812669] Lustre: Skipped 1 previous similar message [ 3324.906682] Lustre: lustre-MDT0001: new disk, initializing [ 3325.127985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3325.144020] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3330.076806] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3334.981321] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3341.415184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3341.738716] Lustre: lustre-OST0000: new disk, initializing [ 3341.743915] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3341.755365] Lustre: 125543:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3343.747828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3343.759224] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3343.877287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3348.150411] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3360.060448] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3360.228654] Lustre: lustre-OST0001: new disk, initializing [ 3360.242691] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3360.250320] Lustre: 126417:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3362.354633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3362.365109] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3362.459706] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3366.967827] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3375.848698] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3378.911813] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3765.086309] Lustre: 132836:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3765.092575] Lustre: 132836:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3765.097077] Lustre: 132836:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3765.102550] Lustre: 132836:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3765.107975] Lustre: 132836:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3765.115971] Lustre: 132836:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4023.376851] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:04:11 (1789452251) [ 4042.228864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4042.246335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4042.269173] Lustre: Skipped 2 previous similar messages [ 4042.279413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4042.287788] Lustre: Skipped 4 previous similar messages [ 4045.416747] Lustre: server umount lustre-MDT0000 complete [ 4048.865677] LustreError: 127239:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4048.895457] LustreError: 127239:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4050.627753] LustreError: 126416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789452282 with bad export cookie 8500165714499934772 [ 4050.635517] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4050.652044] LustreError: 126416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4053.989925] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4054.009349] Lustre: Skipped 3 previous similar messages [ 4054.021476] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4054.035887] Lustre: Skipped 5 previous similar messages [ 4057.255696] Lustre: server umount lustre-MDT0001 complete [ 4067.354596] Lustre: server umount lustre-OST0000 complete [ 4085.217875] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4085.389840] Lustre: server umount lustre-OST0001 complete [ 4091.226568] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 4101.185287] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 4136.579958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4136.910679] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4136.955216] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4137.106719] Lustre: lustre-MDT0000: new disk, initializing [ 4137.211805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4137.222229] Lustre: Skipped 5 previous similar messages [ 4137.246378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4143.214839] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 4154.637115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4154.737355] Lustre: 135894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4154.785176] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4154.788574] Lustre: Skipped 1 previous similar message [ 4154.875209] Lustre: lustre-MDT0001: new disk, initializing [ 4154.994199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4155.007880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4158.833477] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 4168.073147] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4173.376899] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4180.793258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4181.189402] Lustre: lustre-OST0000: new disk, initializing [ 4181.193895] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4181.202312] Lustre: 137864:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4181.315179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4181.321887] Lustre: Skipped 1 previous similar message [ 4183.359902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4183.380755] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4183.548971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4187.217767] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 4196.960327] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4209.252339] Lustre: lustre-MDD0000: changelog on [ 4212.638885] Lustre: lustre-MDD0001: changelog on [ 6073.257693] Lustre: lustre-MDD0001: changelog off [ 6075.254826] Lustre: 139318:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6076.346449] Lustre: lustre-MDD0000: changelog off [ 6083.044953] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6083.052703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6083.054157] LustreError: Skipped 1 previous similar message [ 6083.063197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6083.071910] Lustre: Skipped 2 previous similar messages [ 6083.088105] Lustre: Skipped 3 previous similar messages [ 6086.127295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6091.235322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6091.251601] Lustre: Skipped 2 previous similar messages [ 6097.376900] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6097.643120] Lustre: server umount lustre-OST0000 complete [ 6101.473256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6101.477276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6101.509965] Lustre: Skipped 1 previous similar message [ 6101.530824] Lustre: Skipped 3 previous similar messages [ 6107.972689] Lustre: server umount lustre-MDT0000 complete [ 6111.493890] LustreError: 137862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789454342 with bad export cookie 8500165714500707593 [ 6111.503871] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6111.510541] LustreError: 137862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6111.946763] Lustre: server umount lustre-MDT0001 complete [ 6123.260985] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 6125.934578] Key type lgssc unregistered [ 6126.213155] LNet: 147006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6126.223872] LNetError: 147006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6126.246180] LNet: Removed LNI 192.168.204.134@tcp [ 6127.317186] Key type .llcrypt unregistered [ 6127.322185] Key type ._llcrypt unregistered [ 6149.851437] Key type ._llcrypt registered [ 6149.853902] Key type .llcrypt registered [ 6150.023778] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 6166.783875] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 6167.498898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6167.512836] alg: No test for adler32 (adler32-zlib) [ 6168.516243] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 6168.716507] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 6170.415266] Key type lgssc registered [ 6171.852665] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6210.245105] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6210.295344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6211.651517] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6211.678807] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6211.742450] Lustre: lustre-MDT0000: new disk, initializing [ 6211.831289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6211.861301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6217.078252] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6227.776380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6227.923563] Lustre: 150203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6227.991437] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6227.995031] Lustre: Skipped 1 previous similar message [ 6228.072460] Lustre: lustre-MDT0001: new disk, initializing [ 6228.189779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6228.249381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6228.265845] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6234.110737] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6245.153209] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6251.642623] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6258.258907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6258.643043] Lustre: lustre-OST0000: new disk, initializing [ 6258.649758] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6258.665754] Lustre: 152172:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6258.873862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6265.597082] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6265.990045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6266.009783] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6266.106406] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6276.295974] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6280.818697] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_module obdecho/obdecho [ 6282.241101] LustreError: 153175:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6282.254460] LustreError: 153175:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6283.861388] LustreError: 153271:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6295.009510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6295.015894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6295.033380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6297.058086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6297.059961] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6297.089436] Lustre: Skipped 1 previous similar message [ 6299.533605] Lustre: server umount lustre-MDT0000 complete [ 6304.039914] LustreError: 150196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789454535 with bad export cookie 9071160472383364362 [ 6304.059311] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6304.065861] LustreError: 150196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6304.367659] Lustre: server umount lustre-MDT0001 complete [ 6319.170080] Lustre: server umount lustre-OST0000 complete [ 6341.631697] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 6353.860573] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 6392.007744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6392.281842] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6392.315860] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6392.435875] Lustre: lustre-MDT0000: new disk, initializing [ 6392.547348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6392.574930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6397.889916] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6409.138960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6409.376525] Lustre: 156988:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6409.451690] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6409.456587] Lustre: Skipped 1 previous similar message [ 6409.533302] Lustre: lustre-MDT0001: new disk, initializing [ 6409.626646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6409.668892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6409.685455] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6414.527528] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6425.167656] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6430.965949] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6438.506230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6438.828762] Lustre: lustre-OST0000: new disk, initializing [ 6438.831752] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6438.838620] Lustre: 158957:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6438.938438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6445.101560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6445.118086] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6445.210091] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6445.762199] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6455.042503] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6460.423824] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6460.699674] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6466.849140] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6467.195017] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6471.143413] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6471.152123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6471.165197] LustreError: Skipped 1 previous similar message [ 6471.170477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6471.170486] Lustre: Skipped 2 previous similar messages [ 6471.197272] Lustre: Skipped 1 previous similar message [ 6475.668847] Lustre: server umount lustre-OST0000 complete [ 6481.388519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6481.390173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6481.403894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6481.403902] Lustre: Skipped 1 previous similar message [ 6481.420448] Lustre: Skipped 1 previous similar message [ 6485.700550] Lustre: server umount lustre-MDT0000 complete [ 6486.496264] LustreError: 156995:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6486.509324] LustreError: 156995:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6489.250788] LustreError: 157793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789454720 with bad export cookie 9071160472383365321 [ 6489.258161] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6489.260725] LustreError: 157793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6489.578115] Lustre: server umount lustre-MDT0001 complete [ 6493.653542] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 02:45:22 (1789454722) [ 6495.487343] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6497.691322] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 02:45:26 (1789454726) [ 6507.333783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6508.159750] LustreError: 161373:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6508.364907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6513.642350] LustreError: 161374:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6515.556727] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6517.798547] LustreError: 161373:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6522.851595] LustreError: 161374:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6525.095779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6528.888140] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6536.100756] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6542.574871] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6550.003436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6550.310259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6550.317272] Lustre: Skipped 1 previous similar message [ 6557.084193] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6565.174858] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6576.117723] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.34@tcp (stopping) [ 6576.128720] Lustre: Skipped 1 previous similar message [ 6576.614857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6576.639789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6577.623841] Lustre: server umount lustre-MDT0000 complete [ 6580.198244] LustreError: 161373:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6580.220982] LustreError: 161373:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6581.519076] LustreError: 161354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789454812 with bad export cookie 9071160472383366203 [ 6581.537390] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6585.315888] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6585.329243] Lustre: Skipped 2 previous similar messages [ 6585.333654] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6585.341350] Lustre: Skipped 1 previous similar message [ 6587.957624] Lustre: server umount lustre-MDT0001 complete [ 6594.710876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6608.623175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6608.954950] LustreError: 165215:0:(llog.c:1655:llog_backup()) MGC192.168.204.134@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6608.964753] Lustre: 165215:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.134@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6610.918075] LustreError: 165220:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6610.949097] LustreError: 165220:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6646.650575] LustreError: 165222:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6646.669802] LustreError: 165222:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 26 previous similar messages [ 6669.794185] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6669.797520] Lustre: 165215:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.134@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6710.760500] LustreError: 165238:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6710.766835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6710.781445] LustreError: 165238:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 42 previous similar messages [ 6715.877429] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6716.276812] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6725.319859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6726.113841] LustreError: 148425:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bcf18194 x1876378937067520/t0(0) o250->MGC192.168.204.134@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6731.579396] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6731.777440] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6740.770687] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6742.330238] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6748.615387] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6750.162400] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6767.587347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6767.600485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6767.600512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6767.616774] Lustre: Skipped 3 previous similar messages [ 6773.818387] Lustre: server umount lustre-MDT0000 complete [ 6777.242801] LustreError: 161354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789455008 with bad export cookie 9071160472383372657 [ 6777.249772] LustreError: 161354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6777.253861] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6777.515360] Lustre: server umount lustre-MDT0001 complete [ 6792.041960] Lustre: server umount lustre-OST0000 complete [ 6814.964262] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 6824.102691] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 6858.114345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6858.378703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6858.424915] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6858.513737] Lustre: lustre-MDT0000: new disk, initializing [ 6858.641304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6858.646284] Lustre: Skipped 1 previous similar message [ 6858.669792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6863.277231] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6873.785379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6873.959888] Lustre: 170324:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6874.343656] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6878.877345] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6887.981936] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6893.723668] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6901.399897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6901.621322] Lustre: lustre-OST0000: new disk, initializing [ 6901.624114] Lustre: Skipped 1 previous similar message [ 6901.627311] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6901.631920] Lustre: Skipped 2 previous similar messages [ 6901.636924] Lustre: 172292:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6907.479033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6907.485101] Lustre: Skipped 1 previous similar message [ 6907.492580] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6907.578398] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6907.821824] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6916.237988] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6920.853515] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6921.099986] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6926.273579] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6926.471853] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6928.354654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6928.367380] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6928.378987] Lustre: Skipped 2 previous similar messages [ 6928.383722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6928.403262] Lustre: Skipped 4 previous similar messages [ 6934.458741] Lustre: server umount lustre-OST0000 complete [ 6944.200746] Lustre: server umount lustre-MDT0000 complete [ 6947.497130] LustreError: 170316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789455178 with bad export cookie 9071160472383389492 [ 6947.504901] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6947.513447] LustreError: 170316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6947.953984] Lustre: server umount lustre-MDT0001 complete [ 6956.859619] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 02:53:05 (1789455185) [ 6964.169792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6964.570277] LustreError: 174703:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6964.598116] LustreError: 174703:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 6970.188986] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6980.517985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6986.269890] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6994.727267] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7000.542818] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7008.998481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7017.345110] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7024.832919] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7038.361049] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 02:54:27 (1789455267) [ 7046.625383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7046.629343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7046.634938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7046.634948] Lustre: Skipped 6 previous similar messages [ 7046.641132] LustreError: Skipped 1 previous similar message [ 7046.709671] Lustre: Skipped 4 previous similar messages [ 7048.243516] Lustre: server umount lustre-OST0000 complete [ 7059.724090] Lustre: server umount lustre-MDT0000 complete [ 7064.860202] LustreError: 174684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789455296 with bad export cookie 9071160472383390374 [ 7064.872868] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7064.879791] LustreError: 174684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7065.343849] Lustre: server umount lustre-MDT0001 complete [ 7078.259888] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 7081.364946] Key type lgssc unregistered [ 7081.680973] LNet: 178433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7081.699538] LNetError: 178433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7082.728604] LNet: Removed LNI 192.168.204.134@tcp [ 7083.596576] Key type .llcrypt unregistered [ 7083.599156] Key type ._llcrypt unregistered [ 7103.479547] Key type ._llcrypt registered [ 7103.484027] Key type .llcrypt registered [ 7103.634458] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 7104.541454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7104.618399] alg: No test for adler32 (adler32-zlib) [ 7105.609506] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 7105.802917] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 7107.432695] Key type lgssc registered [ 7108.751165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7117.417850] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7117.451623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7132.959443] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7138.079469] LustreError: 179675:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.134@tcp: failed processing log, type 4: rc = -110 [ 7143.199908] LustreError: 179675:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7143.211974] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7143.219620] LustreError: MGC192.168.204.134@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7143.229671] LustreError: 179675:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7143.239028] LustreError: 179675:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 7143.247203] LustreError: 179675:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 7143.303791] Lustre: server umount lustre-OST0000 complete [ 7143.306608] LustreError: 179675:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7153.721573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7155.216878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7160.009501] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7165.418824] LustreError: 180175:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7165.440066] LustreError: 180175:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7169.941974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7170.200843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7173.788609] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7180.314223] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7185.200462] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7190.949414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7191.169109] LustreError: 181990:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7191.191503] Lustre: 181990:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.134@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7191.385568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7198.015198] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7206.002957] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7209.959691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7209.967311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7209.977275] Lustre: Skipped 2 previous similar messages [ 7210.000710] Lustre: Skipped 2 previous similar messages [ 7214.783877] Lustre: server umount lustre-MDT0000 complete [ 7215.080901] LustreError: 180903:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7215.101479] LustreError: 180903:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7218.979459] LustreError: 180154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789455450 with bad export cookie 17184214965746752371 [ 7218.994419] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7219.380507] Lustre: server umount lustre-MDT0001 complete [ 7235.215681] Lustre: server umount lustre-OST0000 complete [ 7245.336227] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 7261.224843] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 7303.297322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7303.624953] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7303.727949] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7303.860809] Lustre: lustre-MDT0000: new disk, initializing [ 7303.949252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7303.981839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7309.382223] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7322.385730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7322.491627] Lustre: 185538:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7322.559343] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7322.564853] Lustre: Skipped 1 previous similar message [ 7322.641429] Lustre: lustre-MDT0001: new disk, initializing [ 7322.766323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7322.812213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7322.829331] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7327.481594] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7337.126758] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7343.798639] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7351.325788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7351.625913] Lustre: lustre-OST0000: new disk, initializing [ 7351.629222] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7351.636324] Lustre: 187507:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7351.719455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7358.239040] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7359.137927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7359.156653] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7359.331134] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7369.100951] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7374.727671] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7374.909274] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7380.313775] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7380.865191] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7385.056521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7385.058234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7385.069687] LustreError: Skipped 1 previous similar message [ 7385.070633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7385.107951] Lustre: Skipped 1 previous similar message [ 7390.184254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7390.201645] Lustre: Skipped 2 previous similar messages [ 7395.310622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7395.318322] Lustre: Skipped 1 previous similar message [ 7397.855767] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7398.079399] Lustre: server umount lustre-OST0000 complete [ 7404.512483] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7404.522242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7404.551086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7415.776955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7415.782131] Lustre: Skipped 5 previous similar messages [ 7417.311163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7417.736454] Lustre: server umount lustre-MDT0000 complete [ 7420.903500] LustreError: 185546:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7420.923488] LustreError: 185546:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7421.616801] LustreError: 185530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789455653 with bad export cookie 17184214965746753092 [ 7421.624459] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7421.628571] LustreError: 185530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7422.075625] Lustre: server umount lustre-MDT0001 complete [ 7433.372695] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 03:01:01 (1789455661) [ 7455.696776] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 7459.877130] Key type lgssc unregistered [ 7460.327609] LNet: 190468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7460.348520] LNetError: 190468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7460.373970] LNet: Removed LNI 192.168.204.134@tcp [ 7461.716219] Key type .llcrypt unregistered [ 7461.718612] Key type ._llcrypt unregistered [ 7483.801877] Key type ._llcrypt registered [ 7483.803754] Key type .llcrypt registered [ 7483.886001] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 7484.863572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7484.887786] alg: No test for adler32 (adler32-zlib) [ 7486.044120] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 7486.394946] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 7488.175374] Key type lgssc registered [ 7489.686719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7497.929161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7497.948214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7499.375204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7504.105071] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7508.963614] LustreError: 191686:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7508.986038] LustreError: 191686:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7513.709925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7514.089493] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7514.372971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7519.564136] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7527.905140] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7533.466240] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7542.880620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7543.166892] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7543.169087] LustreError: 193550:0:(llog.c:1696:llog_backup()) MGC192.168.204.134@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7543.173036] Lustre: 193550:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.134@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7543.305823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7552.933965] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7561.858781] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7565.792619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7565.806333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7565.831208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7568.864840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7568.865608] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7573.987143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7574.006454] Lustre: Skipped 4 previous similar messages [ 7579.104038] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7579.105148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7579.130509] Lustre: Skipped 2 previous similar messages [ 7579.357946] Lustre: server umount lustre-MDT0000 complete [ 7584.138481] LustreError: 191664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789455815 with bad export cookie 18099432896058554257 [ 7584.141246] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7584.154188] LustreError: 191664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7584.231705] LustreError: 191680:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7584.233389] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7584.272110] Lustre: Skipped 1 previous similar message [ 7584.688889] Lustre: server umount lustre-MDT0001 complete [ 7591.793700] Lustre: server umount lustre-OST0000 complete [ 7602.775412] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 7616.400352] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 7662.701949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7663.040368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7663.090811] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7663.187543] Lustre: lustre-MDT0000: new disk, initializing [ 7663.265264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7663.296141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7668.676926] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7679.712530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7679.824122] Lustre: 197077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7679.857420] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7679.859752] Lustre: Skipped 1 previous similar message [ 7679.862380] LustreError: 197861:0:(llog.c:1696:llog_backup()) MGC192.168.204.134@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7679.868901] LustreError: 197861:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7679.872602] Lustre: 197861:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.134@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7679.888377] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7679.897217] Lustre: Skipped 1 previous similar message [ 7679.993327] Lustre: lustre-MDT0001: new disk, initializing [ 7680.110423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7680.160665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7680.182986] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7685.348683] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7695.392428] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7701.460349] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7709.458666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7709.790820] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7709.792616] Lustre: Skipped 1 previous similar message [ 7709.797882] LustreError: 199022:0:(llog.c:1696:llog_backup()) MGC192.168.204.134@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7709.809068] LustreError: 199022:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7709.818781] Lustre: 199022:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.134@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7709.835663] Lustre: 199022:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7709.867969] Lustre: lustre-OST0000: new disk, initializing [ 7709.871702] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7709.877551] Lustre: 199046:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7709.964924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7716.006040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7716.017279] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7716.186847] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7716.535140] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7727.195498] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7733.818121] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7734.147522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7740.145928] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7740.417306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7747.041079] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7747.057853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7747.063505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7747.079714] Lustre: Skipped 1 previous similar message [ 7747.094592] Lustre: Skipped 2 previous similar messages [ 7757.279152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7757.480861] Lustre: server umount lustre-OST0000 complete [ 7762.404825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7762.431055] Lustre: Skipped 1 previous similar message [ 7767.520806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7767.529085] Lustre: Skipped 7 previous similar messages [ 7768.316558] Lustre: server umount lustre-MDT0000 complete [ 7772.352920] LustreError: 197071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789456003 with bad export cookie 18099432896058554978 [ 7772.366264] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7772.374584] LustreError: 197071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7772.873269] Lustre: server umount lustre-MDT0001 complete [ 7782.727332] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 03:06:51 (1789456011) [ 7805.421569] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 7817.774681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7818.238451] LustreError: 202201:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7818.277988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7823.345502] LustreError: 202202:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7823.804770] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7828.461724] LustreError: 202201:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7837.373084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7843.929573] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7847.133766] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7856.710784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7857.147988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7857.155596] Lustre: Skipped 1 previous similar message [ 7865.502490] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7875.087951] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7875.337607] Lustre: lustre-OST0001: new disk, initializing [ 7875.342527] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7875.352598] Lustre: 204510:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7881.780429] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7881.831514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7881.837233] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7881.952973] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7892.354603] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7902.347901] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7904.742306] Lustre: DEBUG MARKER: ADD OST3 [ 7910.230020] loop: module loaded [ 7910.294856] loop0: detected capacity change from 0 to 204800000 [ 7910.583576] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7910.602734] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7910.615101] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7917.240699] loop0: detected capacity change from 0 to 204800000 [ 7917.856569] Lustre: DEBUG MARKER: STOP OST3 [ 7924.273530] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7924.527451] Lustre: lustre-OST0003: new disk, initializing [ 7924.533577] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7924.547406] Lustre: 206553:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7924.692456] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7924.696718] Lustre: Skipped 1 previous similar message [ 7926.370397] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7926.531364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7926.540770] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7926.610720] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7928.269146] Lustre: Failing over lustre-OST0003 [ 7928.423107] LustreError: 206693:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7928.757613] Lustre: server umount lustre-OST0003 complete [ 7930.789795] LustreError: 203957:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7930.810540] LustreError: 203957:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7931.876322] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7931.896472] LustreError: Skipped 1 previous similar message [ 7931.914852] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7931.928529] Lustre: Skipped 1 previous similar message [ 7932.912633] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7942.399820] Lustre: DEBUG MARKER: START OST3 again [ 7947.247383] LustreError: 203689:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7947.263834] LustreError: 203689:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 7950.486755] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7950.874412] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7951.235231] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7952.626483] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7952.629442] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7958.465766] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7980.207817] Lustre: Failing over lustre-OST0003 [ 7980.369693] Lustre: server umount lustre-OST0003 complete [ 7981.551126] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7981.562463] LustreError: 203691:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7981.579072] LustreError: 203691:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8001.505531] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8001.522869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8013.279113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8013.546929] Lustre: server umount lustre-MDT0000 complete [ 8017.179406] LustreError: 202182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789456248 with bad export cookie 18099432896058555860 [ 8017.194833] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8017.205217] LustreError: 202182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8017.378026] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8017.387980] Lustre: Skipped 6 previous similar messages [ 8023.793414] Lustre: server umount lustre-MDT0001 complete [ 8027.972417] Lustre: server umount lustre-OST0000 complete [ 8032.529405] Lustre: server umount lustre-OST0001 complete [ 8040.854867] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 8051.323931] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 8092.034115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8092.313166] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8092.388357] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8092.501623] Lustre: lustre-MDT0000: new disk, initializing [ 8092.567953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8092.580343] Lustre: Skipped 1 previous similar message [ 8092.615372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8098.473735] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8108.947491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8109.073277] Lustre: 211974:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8109.170503] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8109.177821] Lustre: Skipped 1 previous similar message [ 8109.241951] Lustre: lustre-MDT0001: new disk, initializing [ 8109.387711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8109.396037] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8114.580447] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8124.518639] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8130.438812] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8138.054049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8138.490624] Lustre: 213947:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8140.567437] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8140.688950] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8145.298699] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8156.310319] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8161.710898] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8161.937740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8166.783737] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8166.998775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8171.495670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8171.498098] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8171.502693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8171.502702] Lustre: Skipped 16 previous similar messages [ 8171.508393] LustreError: Skipped 1 previous similar message [ 8171.535734] Lustre: Skipped 1 previous similar message [ 8183.263129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8183.410247] Lustre: server umount lustre-OST0000 complete [ 8191.456319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8193.608806] Lustre: server umount lustre-MDT0000 complete [ 8197.089416] LustreError: 211984:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8197.121148] LustreError: 211984:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [ 8197.548492] LustreError: 212770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789456428 with bad export cookie 18099432896058558149 [ 8197.550395] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8197.562575] LustreError: 212770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8197.989754] Lustre: server umount lustre-MDT0001 complete [ 8202.183365] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 03:13:50 (1789456430) [ 8219.791259] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 8230.168703] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 8267.665437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8267.897774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8267.926365] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8267.930827] Lustre: Skipped 1 previous similar message [ 8268.051519] Lustre: lustre-MDT0000: new disk, initializing [ 8268.055589] Lustre: Skipped 1 previous similar message [ 8268.174774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8268.182444] Lustre: Skipped 1 previous similar message [ 8272.773742] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8283.804222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8283.960905] Lustre: 218560:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8284.326940] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8288.715742] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8297.665669] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8303.350272] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8311.632438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8311.968557] Lustre: 220530:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8313.210252] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8319.669324] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8329.581157] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8334.927736] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8335.230202] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8339.784377] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8340.130741] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8344.033046] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8344.047583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8344.055745] LustreError: Skipped 1 previous similar message [ 8344.065684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8344.075371] Lustre: Skipped 3 previous similar messages [ 8344.085758] Lustre: Skipped 8 previous similar messages [ 8348.499315] Lustre: server umount lustre-OST0000 complete [ 8358.799462] Lustre: server umount lustre-MDT0000 complete [ 8359.393496] LustreError: 218570:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8359.412458] LustreError: 218570:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8363.284447] LustreError: 218551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789456594 with bad export cookie 18099432896058559038 [ 8363.285549] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8363.296252] LustreError: 218551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8363.677924] Lustre: server umount lustre-MDT0001 complete [ 8372.302259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8372.874656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8372.883603] Lustre: Skipped 5 previous similar messages [ 8377.718627] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8386.452383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8391.372437] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8398.863880] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8403.911140] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8410.731993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8417.012918] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8424.203363] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8433.638470] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8439.862595] Lustre: server umount lustre-OST0000 complete [ 8450.658705] Lustre: server umount lustre-MDT0000 complete [ 8454.715829] LustreError: 222725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789456686 with bad export cookie 18099432896058559920 [ 8454.735361] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8454.740206] LustreError: 222725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8455.137444] Lustre: server umount lustre-MDT0001 complete [ 8564.886467] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 8567.534436] Key type lgssc unregistered [ 8567.820613] LNet: 226411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8567.824617] LNetError: 226411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8567.835354] LNet: Removed LNI 192.168.204.134@tcp [ 8568.539175] Key type .llcrypt unregistered [ 8568.542229] Key type ._llcrypt unregistered [ 8592.980942] Key type ._llcrypt registered [ 8592.983766] Key type .llcrypt registered [ 8593.071479] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 8608.890745] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 8610.459555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8610.475847] alg: No test for adler32 (adler32-zlib) [ 8612.151671] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 8612.503726] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 8614.231232] Key type lgssc registered [ 8615.413573] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8651.530321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8651.552567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8652.749368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8652.796650] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8652.884819] Lustre: lustre-MDT0000: new disk, initializing [ 8652.988837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8653.010377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8657.751928] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8668.811293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8669.016406] Lustre: 229760:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8669.080516] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8669.087058] Lustre: Skipped 1 previous similar message [ 8669.158870] Lustre: lustre-MDT0001: new disk, initializing [ 8669.241185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8669.284739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8669.298423] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8674.109983] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8683.644742] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8689.425058] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8696.493485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8696.849206] Lustre: lustre-OST0000: new disk, initializing [ 8696.853665] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8696.859886] Lustre: 231729:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8697.004302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8703.752432] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8704.121540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8704.137382] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8704.251870] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8714.902615] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8720.320325] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8720.579651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8725.798126] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8726.167987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8730.080948] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8730.086334] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8730.098234] LustreError: Skipped 1 previous similar message [ 8730.099447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8730.152927] Lustre: Skipped 1 previous similar message [ 8734.548909] Lustre: server umount lustre-OST0000 complete [ 8740.320938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8740.328538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8740.334783] Lustre: Skipped 1 previous similar message [ 8740.340792] Lustre: Skipped 1 previous similar message [ 8745.525672] Lustre: server umount lustre-MDT0000 complete [ 8749.402021] LustreError: 229753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789456980 with bad export cookie 6909137856756486844 [ 8749.413130] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8749.416135] LustreError: 229753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8749.836892] Lustre: server umount lustre-MDT0001 complete [ 8769.840300] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 8781.418268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8781.843761] LustreError: 234665:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8781.869717] LustreError: 234665:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8781.922244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8786.778719] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8786.914469] LustreError: 234666:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8792.034616] LustreError: 234665:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8795.320385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8795.625418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8799.323307] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8801.457990] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8807.745753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8808.493240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8819.216878] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8836.736906] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8837.035320] Lustre: lustre-OST0001: new disk, initializing [ 8837.038750] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8837.044864] Lustre: 236975:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8837.204191] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8843.939184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8843.966724] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8844.236287] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8846.548693] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8860.244990] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8866.827193] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8881.629692] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 03:25:10 (1789457110) [ 8885.216800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8885.229823] Lustre: Skipped 3 previous similar messages [ 8885.230145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8885.253135] Lustre: Skipped 4 previous similar messages [ 8890.301266] Lustre: server umount lustre-MDT0000 complete [ 8890.338908] LustreError: 234665:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8890.356336] LustreError: 234665:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8894.310219] LustreError: 234648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789457125 with bad export cookie 6909137856756487726 [ 8894.310912] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8894.316630] LustreError: 234648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8894.686291] Lustre: server umount lustre-MDT0001 complete [ 8908.374762] Lustre: server umount lustre-OST0000 complete [ 8911.327808] Lustre: 227980:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789457126/real 1789457126] req@ffff8f25070fea00 x1876381498724736/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789457142 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8911.355141] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8913.001903] Lustre: server umount lustre-OST0001 complete [ 8920.601289] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 8931.156790] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 8966.105970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8966.286377] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8966.322577] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8966.427564] Lustre: lustre-MDT0000: new disk, initializing [ 8966.543183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8966.564524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8971.072621] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8982.178348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8982.293081] Lustre: 240925:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8982.395186] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8982.407908] Lustre: Skipped 1 previous similar message [ 8982.522928] Lustre: lustre-MDT0001: new disk, initializing [ 8982.734992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8982.752853] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8987.694203] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8996.935544] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9002.638611] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9010.298973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9010.625129] Lustre: lustre-OST0000: new disk, initializing [ 9010.633837] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9010.646491] Lustre: 242894:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9012.365745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9012.376759] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9012.505422] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9018.145951] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9028.805191] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9034.737526] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9034.950525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9040.394320] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9040.588471] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9043.431087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9043.433225] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9043.436863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9043.438631] Lustre: Skipped 1 previous similar message [ 9043.457303] LustreError: Skipped 1 previous similar message [ 9048.546422] LustreError: 242988:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9048.722745] Lustre: server umount lustre-OST0000 complete [ 9053.665892] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9053.677214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9053.686774] Lustre: Skipped 1 previous similar message [ 9053.706400] Lustre: Skipped 2 previous similar messages [ 9058.965457] Lustre: server umount lustre-MDT0000 complete [ 9062.923597] LustreError: 240917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789457294 with bad export cookie 6909137856756489861 [ 9062.931475] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9062.941163] LustreError: 240917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9063.425522] Lustre: server umount lustre-MDT0001 complete [ 9073.513115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9074.023345] LustreError: 245112:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9074.045314] LustreError: 245112:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9074.095465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9074.104729] Lustre: Skipped 2 previous similar messages [ 9079.283451] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9088.761298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9094.565840] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9102.506768] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9109.252511] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9116.785601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9124.852067] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9133.167160] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9282.530070] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9282.530609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9282.561456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9282.572738] Lustre: Skipped 3 previous similar messages [ 9286.504235] Lustre: server umount lustre-OST0000 complete [ 9296.473287] Lustre: server umount lustre-MDT0000 complete [ 9297.891918] LustreError: 247627:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9297.913439] LustreError: 247627:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9299.837907] LustreError: 245095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789457531 with bad export cookie 6909137856756490743 [ 9299.840469] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9299.849120] LustreError: 245095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9300.173623] Lustre: server umount lustre-MDT0001 complete [ 9399.306091] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 9402.281799] Key type lgssc unregistered [ 9402.563671] LNet: 248721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9402.572551] LNetError: 248721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9402.589019] LNet: Removed LNI 192.168.204.134@tcp [ 9403.540147] Key type .llcrypt unregistered [ 9403.542503] Key type ._llcrypt unregistered [ 9429.972278] Key type ._llcrypt registered [ 9429.975642] Key type .llcrypt registered [ 9430.120267] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 9447.555380] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 9449.011094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9449.091746] alg: No test for adler32 (adler32-zlib) [ 9450.185259] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 9450.464396] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 9452.223199] Key type lgssc registered [ 9453.253594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9506.917110] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 9517.234856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9517.245369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9518.675736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9518.715435] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9518.855261] Lustre: lustre-MDT0000: new disk, initializing [ 9518.935350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9518.952263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9523.847884] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9534.875091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9534.990893] Lustre: 252817:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9535.050387] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9535.056092] Lustre: Skipped 1 previous similar message [ 9535.135389] Lustre: lustre-MDT0001: new disk, initializing [ 9535.222993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9535.259243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9535.275328] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9540.548287] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9544.872650] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9550.692275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9550.852741] Lustre: lustre-OST0000: new disk, initializing [ 9550.856268] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9550.861197] Lustre: 254452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9550.976224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9553.748472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9553.769483] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9553.914091] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9556.568666] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9566.231452] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9566.438631] Lustre: lustre-OST0001: new disk, initializing [ 9566.447739] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9566.460619] Lustre: 255323:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9566.553558] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9572.365168] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9574.952812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9574.965144] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9575.108499] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9583.405305] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9590.683183] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9600.995837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9601.006266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9601.016515] Lustre: Skipped 3 previous similar messages [ 9601.035634] Lustre: Skipped 3 previous similar messages [ 9606.115747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9606.815692] Lustre: server umount lustre-MDT0000 complete [ 9610.104681] LustreError: 253614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789457841 with bad export cookie 14478261346363382771 [ 9610.105381] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9610.110464] LustreError: 253614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9610.385226] Lustre: server umount lustre-MDT0001 complete [ 9624.084706] Lustre: server umount lustre-OST0000 complete [ 9627.040676] Lustre: 250295:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789457842/real 1789457842] req@ffff8f240ecf3480 x1876382377387904/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789457858 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9627.066612] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9627.403230] Lustre: server umount lustre-OST0001 complete [ 9641.753777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9652.390237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9660.901767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9670.562165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9689.368721] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 9699.914991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9699.962180] Lustre: lustre-MDT0000: reset Object Index mappings [ 9700.530318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9705.022693] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9705.972765] LustreError: 260328:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9705.993336] LustreError: 260328:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9710.120945] LustreError: 260329:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9713.569464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9713.596717] Lustre: lustre-MDT0001: reset Object Index mappings [ 9714.106406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9719.048315] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9722.210505] Lustre: 261469:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9728.371334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9734.179360] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9738.995815] LustreError: 262410:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9742.977170] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9743.214803] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9743.227977] Lustre: Skipped 1 previous similar message [ 9745.281698] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9745.287510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9749.631928] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9757.145071] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9760.846620] Lustre: 263343:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9766.769276] Lustre: server umount lustre-MDT0000 complete [ 9770.470191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9770.475809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9770.488469] LustreError: 260329:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9770.919538] LustreError: 260308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789458002 with bad export cookie 14478261346363389561 [ 9770.937250] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9770.943622] LustreError: 260308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9771.000312] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9771.008361] Lustre: Skipped 3 previous similar messages [ 9771.615323] Lustre: server umount lustre-MDT0001 complete [ 9777.676126] Lustre: server umount lustre-OST0000 complete [ 9781.522635] Lustre: server umount lustre-OST0001 complete [ 9802.285598] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 03:40:31 (1789458031) [ 9822.486181] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 9834.629454] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 9879.207680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9879.630943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9879.705316] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9879.825837] Lustre: lustre-MDT0000: new disk, initializing [ 9879.934391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9879.952226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9886.795692] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9898.148494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9898.260806] Lustre: 267601:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9898.300671] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9898.304863] Lustre: Skipped 1 previous similar message [ 9898.372721] Lustre: lustre-MDT0001: new disk, initializing [ 9898.508404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9898.519968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9903.045155] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9913.885696] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9919.657207] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9927.122573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9927.481472] Lustre: lustre-OST0000: new disk, initializing [ 9927.488644] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9927.497514] Lustre: 269569:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9929.033117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9929.044977] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9929.174845] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9933.485647] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 9942.678564] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9946.752027] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9947.033850] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9950.652975] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9950.917977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9954.786494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9954.788918] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9954.793988] LustreError: Skipped 1 previous similar message [ 9954.805389] Lustre: Skipped 4 previous similar messages [ 9954.812551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9954.822693] Lustre: Skipped 2 previous similar messages [ 9958.630106] Lustre: server umount lustre-OST0000 complete [ 9965.026990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9965.028684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9965.049330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9965.066491] Lustre: Skipped 1 previous similar message [ 9975.265088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9975.277089] Lustre: Skipped 3 previous similar messages [ 9976.287147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9976.498679] Lustre: server umount lustre-MDT0000 complete [ 9980.394294] LustreError: 267614:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9980.418282] LustreError: 267614:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9980.676942] LustreError: 267593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789458212 with bad export cookie 14478261346363391654 [ 9980.687581] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9980.693806] LustreError: 267593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9981.076889] Lustre: server umount lustre-MDT0001 complete [10001.815192] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [10012.697366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10012.966210] LustreError: 272504:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10013.005980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10013.009833] Lustre: Skipped 2 previous similar messages [10017.817797] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10026.942492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10031.727382] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10034.943578] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10041.813236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10048.843835] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10056.788426] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10056.950697] Lustre: lustre-OST0001: new disk, initializing [10056.957595] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10056.964158] Lustre: 274811:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10058.670111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10058.677328] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10058.800838] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10063.196538] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10073.043748] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10076.573814] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10081.510175] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10081.554558] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10092.622013] Lustre: server umount lustre-MDT0000 complete [10094.050039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10094.068760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10094.081568] Lustre: Skipped 1 previous similar message [10094.088139] LustreError: 272504:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10094.104956] LustreError: 272504:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10096.631623] LustreError: 272484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789458328 with bad export cookie 14478261346363392536 [10096.643936] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10096.649430] LustreError: 272484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10097.007606] Lustre: server umount lustre-MDT0001 complete [10110.911659] Lustre: server umount lustre-OST0000 complete [10125.737242] Lustre: server umount lustre-OST0001 complete [10144.892576] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [10156.830947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10157.158337] LustreError: 278158:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10157.181572] LustreError: 278158:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10157.221677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10157.225751] Lustre: Skipped 3 previous similar messages [10161.592777] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10170.507795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10175.822727] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10179.160859] Lustre: 279300:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10186.116303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10191.956784] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10201.636378] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10203.087682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10203.104707] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10209.560601] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10219.178220] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10223.199828] Lustre: 281178:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10229.211221] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10229.252352] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10242.084848] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 03:47:50 (1789458470) [10257.895618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10257.914218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10257.933179] Lustre: Skipped 3 previous similar messages [10257.940304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10263.905711] Lustre: server umount lustre-MDT0000 complete [10264.546110] LustreError: 278159:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10264.576939] LustreError: 278159:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [10267.433497] LustreError: 278140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789458498 with bad export cookie 14478261346363398934 [10267.449516] LustreError: 278140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10267.461107] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10273.848198] Lustre: server umount lustre-MDT0001 complete [10283.229424] Lustre: server umount lustre-OST0000 complete [10287.325247] Lustre: server umount lustre-OST0001 complete [10294.520676] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [10306.028735] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [10355.994322] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [10367.139822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10367.364350] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10367.394623] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10367.480801] Lustre: lustre-MDT0000: new disk, initializing [10367.602212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10372.482563] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10383.906388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10384.067332] Lustre: 285704:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10384.401426] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10389.392732] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10401.446698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10401.760355] Lustre: 287341:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10403.654290] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10409.168507] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10420.554571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10420.799832] Lustre: 288212:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10420.933470] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10420.939670] Lustre: Skipped 6 previous similar messages [10422.684247] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10422.689679] Lustre: Skipped 1 previous similar message [10422.887719] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10428.036775] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10438.339214] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10442.257806] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10442.270146] Lustre: Skipped 1 previous similar message [10456.808486] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 03:51:25 (1789458685) [10461.156398] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10461.163758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10461.174173] Lustre: Skipped 5 previous similar messages [10461.189366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10461.201618] Lustre: Skipped 8 previous similar messages [10465.373582] Lustre: server umount lustre-MDT0000 complete [10468.836310] LustreError: 286561:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10468.866745] LustreError: 286561:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [10469.577174] LustreError: 285698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789458701 with bad export cookie 14478261346363406872 [10469.584528] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10469.592589] LustreError: 285698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10470.001884] Lustre: server umount lustre-MDT0001 complete [10484.323592] Lustre: server umount lustre-OST0000 complete [10499.267242] Lustre: server umount lustre-OST0001 complete [10507.422550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10512.331518] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10521.862392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10527.291295] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10534.284923] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10540.920875] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10548.542369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10555.677678] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10563.340630] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10582.293793] loop1: detected capacity change from 0 to 204800000 [10582.577820] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10582.588875] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10582.598632] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10586.912411] loop1: detected capacity change from 0 to 204800000 [10591.953691] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10592.023240] LustreError: 290445:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs5656: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10592.036237] LustreError: 290445:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs5656 [10592.042616] LustreError: 290445:0:(mgs_handler.c:517:mgs_target_reg()) fs5656-MDT0001: Error checking target index: rc = -13 [10592.050170] LustreError: fs5656-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10592.059812] LustreError: 294174:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10592.065847] LustreError: 294174:0:(tgt_mount.c:2128:server_put_super()) no obd fs5656-MDT0001 [10592.072114] LustreError: 294174:0:(tgt_mount.c:116:server_deregister_mount()) fs5656-MDT0001 not registered [10592.116682] Lustre: server umount fs5656-MDT0001 complete [10592.120174] LustreError: 294174:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10601.740432] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10602.036063] Lustre: Setting parameter fs5656-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs5656-MDT0001 [10602.138762] Lustre: srv-fs5656-MDT0001: No data found on store. Initialize space. [10602.157340] Lustre: Skipped 4 previous similar messages [10602.262416] Lustre: fs5656-MDT0001: new disk, initializing [10602.270228] Lustre: Skipped 3 previous similar messages [10602.445446] Lustre: srv-fs5656-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10602.459524] Lustre: 294641:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5656-MDT0001-mdtlov: can't create new llog: rc = -115 [10602.478322] LustreError: 294641:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs5656-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10607.521666] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10614.999856] Lustre: server umount fs5656-MDT0001 complete [10629.520867] loop1: detected capacity change from 0 to 204800000 [10633.206383] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10635.694786] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10636.331520] Lustre: 296207:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5656-MDT0001-mdtlov: can't create new llog: rc = -115 [10636.346695] LustreError: 296207:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs5656-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10641.880804] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10646.832285] Lustre: server umount fs5656-MDT0001 complete [10658.791710] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10658.797616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10658.803186] Lustre: Skipped 5 previous similar messages [10658.811923] Lustre: Skipped 6 previous similar messages [10663.537777] Lustre: server umount lustre-MDT0000 complete [10671.848882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10672.008715] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10676.898700] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10677.747270] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10684.268315] loop1: detected capacity change from 0 to 204800000 [10689.711454] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10690.091334] Lustre: 298653:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5656-MDT0001-mdtlov: can't create new llog: rc = -115 [10690.105065] LustreError: 298653:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs5656-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10695.066255] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10699.635690] Lustre: server umount fs5656-MDT0001 complete [10720.225251] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10720.233288] LustreError: Skipped 1 previous similar message [10726.422562] Lustre: server umount lustre-OST0000 complete [10736.638752] Lustre: server umount lustre-MDT0000 complete [10739.170170] LustreError: 290452:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10739.202345] LustreError: 290452:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 107 previous similar messages [10741.419931] LustreError: 290438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789458972 with bad export cookie 14478261346363409553 [10741.441052] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10741.449508] LustreError: 290438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10741.840813] Lustre: server umount lustre-MDT0001 complete [10754.070050] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [10756.890577] Key type lgssc unregistered [10757.239362] LNet: 300998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10757.244761] LNetError: 300998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10757.262775] LNet: Removed LNI 192.168.204.134@tcp [10758.108215] Key type .llcrypt unregistered [10758.112318] Key type ._llcrypt unregistered [10778.699194] Key type ._llcrypt registered [10778.704314] Key type .llcrypt registered [10778.823751] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [10780.479176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10780.502983] alg: No test for adler32 (adler32-zlib) [10781.761323] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [10782.089393] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [10783.863276] Key type lgssc registered [10785.212754] Lustre: Echo OBD driver; http://www.lustre.org/ [10793.706676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10793.730894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10795.262173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10800.231269] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10804.711964] LustreError: 302271:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10804.744455] LustreError: 302271:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10809.569439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10809.905274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10814.591519] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10822.706911] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10828.591845] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10835.603527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10836.146926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10840.238098] LustreError: 304098:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10843.413141] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10845.666764] LustreError: 304101:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10850.787109] LustreError: 304098:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10851.995638] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10854.888136] LustreError: 304306:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10854.925106] LustreError: 304306:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10864.016749] LustreError: 304306:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10864.038262] LustreError: 304306:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10873.623078] loop1: detected capacity change from 0 to 204800000 [10873.839637] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10873.853890] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10873.863797] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10879.727158] loop1: detected capacity change from 0 to 204800000 [10880.490493] LustreError: 304306:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10880.515294] LustreError: 304306:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [10885.900663] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10886.115720] Lustre: fs4597-OST0000: new disk, initializing [10886.120364] Lustre: srv-fs4597-OST0000: No data found on store. Initialize space. [10886.132875] Lustre: 306006:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10886.246571] Lustre: fs4597-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10893.293316] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10901.556632] Lustre: server umount fs4597-OST0000 complete [10914.684847] loop1: detected capacity change from 0 to 204800000 [10915.198525] LustreError: 304101:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10915.222151] LustreError: 304101:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 18 previous similar messages [10920.557677] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10920.834657] Lustre: fs4597-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10921.454941] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10927.817490] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10936.818413] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10937.648040] loop2: detected capacity change from 0 to 204800000 [10937.944491] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10937.963877] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10937.986742] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10944.614600] loop2: detected capacity change from 0 to 204800000 [10951.122739] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10951.199128] LustreError: fs4597-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10951.232986] LustreError: fs4597-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10951.238062] LustreError: 309046:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10951.244140] LustreError: 309046:0:(tgt_mount.c:2128:server_put_super()) no obd fs4597-OST0001 [10951.253155] LustreError: 309046:0:(tgt_mount.c:116:server_deregister_mount()) fs4597-OST0001 not registered [10951.304103] Lustre: server umount fs4597-OST0001 complete [10951.307252] LustreError: 309046:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10962.260567] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10962.563777] Lustre: fs4597-OST0001: new disk, initializing [10962.572957] Lustre: srv-fs4597-OST0001: No data found on store. Initialize space. [10962.583285] Lustre: 309597:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10962.685373] Lustre: fs4597-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10970.775626] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [10980.718737] Lustre: server umount fs4597-OST0000 complete [10981.774926] LustreError: 304306:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10981.815809] LustreError: 304306:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [10991.147590] Lustre: server umount fs4597-OST0001 complete [11004.499409] loop1: detected capacity change from 0 to 204800000 [11010.045641] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11010.267588] Lustre: fs4597-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11017.786664] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11024.897917] loop2: detected capacity change from 0 to 204800000 [11030.766175] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11038.229414] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11048.470440] Lustre: server umount fs4597-OST0000 complete [11059.538890] Lustre: server umount fs4597-OST0001 complete [11075.052887] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11075.059562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11075.066834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11075.081176] Lustre: Skipped 1 previous similar message [11080.046959] Lustre: server umount lustre-OST0000 complete [11085.281377] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11085.295800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11085.307528] Lustre: Skipped 1 previous similar message [11085.322757] Lustre: Skipped 2 previous similar messages [11090.400550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11090.409366] Lustre: Skipped 1 previous similar message [11095.522882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11095.527834] Lustre: Skipped 1 previous similar message [11099.103601] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11099.368152] Lustre: server umount lustre-MDT0000 complete [11103.732707] LustreError: 302251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789459335 with bad export cookie 7617532207350940357 [11103.753568] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11104.269180] Lustre: server umount lustre-MDT0001 complete [11116.687711] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [11119.851771] Key type lgssc unregistered [11120.202947] LNet: 315535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11120.207686] LNetError: 315535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11121.254293] LNet: Removed LNI 192.168.204.134@tcp [11122.200162] Key type .llcrypt unregistered [11122.202299] Key type ._llcrypt unregistered [11157.338802] Key type ._llcrypt registered [11157.341581] Key type .llcrypt registered [11157.449634] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [11172.687978] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [11173.659718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11173.787079] alg: No test for adler32 (adler32-zlib) [11174.893098] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [11175.131576] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [11176.815488] Key type lgssc registered [11178.167537] Lustre: Echo OBD driver; http://www.lustre.org/ [11218.398420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11218.442381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11219.773634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11219.821744] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11219.930957] Lustre: lustre-MDT0000: new disk, initializing [11220.063362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11220.130314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11226.114194] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11238.494093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11238.645936] Lustre: 319395:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11238.714524] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11238.717609] Lustre: Skipped 1 previous similar message [11238.784932] Lustre: lustre-MDT0001: new disk, initializing [11238.897362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11238.967489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11238.980052] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11244.545177] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11255.003625] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11261.776329] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11270.669452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11270.984447] Lustre: lustre-OST0000: new disk, initializing [11270.987898] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11270.997102] Lustre: 321366:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11271.148256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11279.138450] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11281.486779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11281.494909] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11281.689955] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11292.264750] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11294.819775] LustreError: 317567:0:(lmv_obd.c:211:lmv_notify()) activation of lustre-MDT0000_UUID failed: -22 [11294.896241] Lustre: Mounted lustre-client - version 2.17.58_40_gc6c2479 [11296.345378] Lustre: Unmounted lustre-client [11307.488746] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11307.507326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11307.522738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11307.535151] Lustre: Skipped 1 previous similar message [11310.570193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11310.580595] Lustre: Skipped 1 previous similar message [11313.095877] Lustre: server umount lustre-OST0000 complete [11320.804136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11320.805490] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11320.805811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11320.833619] Lustre: Skipped 2 previous similar messages [11320.858548] LustreError: Skipped 1 previous similar message [11323.978927] Lustre: server umount lustre-MDT0000 complete [11328.888251] LustreError: 319389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789459560 with bad export cookie 14590203482981948859 [11328.906356] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11329.269226] Lustre: server umount lustre-MDT0001 complete [11343.275588] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [11345.120420] LNet: 1 local NIs in recovery (showing 1): 192.168.204.134@tcp [11346.973190] Key type lgssc unregistered [11347.288665] LNet: 323525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11347.295797] LNetError: 323525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11425.127153] LNet: Removed LNI 192.168.204.134@tcp [11426.327038] Key type .llcrypt unregistered [11426.329908] Key type ._llcrypt unregistered [11449.105408] Key type ._llcrypt registered [11449.109356] Key type .llcrypt registered [11449.231495] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [11468.496102] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [11470.298850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11470.332075] alg: No test for adler32 (adler32-zlib) [11471.491674] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [11472.050094] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [11473.951249] Key type lgssc registered [11475.507225] Lustre: Echo OBD driver; http://www.lustre.org/ [11508.980945] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 04:08:57 (1789459737) [11516.796810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11516.816812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11518.187640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11518.225701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11518.387648] Lustre: lustre-MDT0000: new disk, initializing [11518.465738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11518.485371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11524.023467] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11536.431401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11536.587826] Lustre: 326746:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11536.654768] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11536.658305] Lustre: Skipped 1 previous similar message [11536.743912] Lustre: lustre-MDT0001: new disk, initializing [11536.867345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11536.912922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11536.927194] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11542.647498] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11553.342267] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11559.978826] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11567.733853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11568.165994] Lustre: lustre-OST0000: new disk, initializing [11568.171609] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11568.179719] Lustre: 328713:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11568.307084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11574.371416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11574.383082] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11574.541600] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11575.570595] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11585.451869] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11600.357247] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11600.360535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11600.379984] LustreError: Skipped 1 previous similar message [11600.384889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11600.392749] Lustre: Skipped 1 previous similar message [11600.415131] Lustre: Skipped 1 previous similar message [11605.484776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11605.494059] Lustre: Skipped 1 previous similar message [11605.738838] Lustre: server umount lustre-OST0000 complete [11610.593619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11610.615929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11610.626040] Lustre: Skipped 1 previous similar message [11615.899562] Lustre: server umount lustre-MDT0000 complete [11619.837187] LustreError: 326737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789459851 with bad export cookie 11200675414972130076 [11619.850751] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11619.855344] LustreError: 326737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11620.299984] Lustre: server umount lustre-MDT0001 complete [11631.690437] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [11635.294173] Key type lgssc unregistered [11635.681979] LNet: 330619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11635.690537] LNetError: 330619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11635.715913] LNet: Removed LNI 192.168.204.134@tcp [11636.602116] Key type .llcrypt unregistered [11636.606676] Key type ._llcrypt unregistered [11660.301783] Key type ._llcrypt registered [11660.309159] Key type .llcrypt registered [11660.443965] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [11661.788367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11661.888950] alg: No test for adler32 (adler32-zlib) [11663.298356] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [11663.647301] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [11665.417163] Key type lgssc registered [11666.898375] Lustre: Echo OBD driver; http://www.lustre.org/ [11675.165852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11675.196740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11676.717272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11681.558272] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11686.880423] LustreError: 331889:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11686.906408] LustreError: 331889:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11692.008179] LustreError: 331888:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11692.474136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11692.729395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11697.976266] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11706.161702] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11712.581199] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11719.177497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11719.546044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11725.927688] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11732.780848] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11746.008891] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11747.873834] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11754.467137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11754.469644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11754.475418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11754.496731] Lustre: Skipped 1 previous similar message [11755.922695] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.34@tcp (stopping) [11755.932314] Lustre: Skipped 2 previous similar messages [11758.794620] Lustre: server umount lustre-MDT0000 complete [11759.584758] LustreError: 332617:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11759.600932] LustreError: 332617:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11763.458749] LustreError: 331870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789459994 with bad export cookie 5088916396791690508 [11763.464735] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11763.467925] LustreError: 331870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11764.706376] LustreError: 331885:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11764.706616] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11764.726836] LustreError: 331885:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11764.740437] Lustre: Skipped 1 previous similar message [11764.751511] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11768.748167] LustreError: 331884:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11768.753862] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11768.782572] LustreError: 331884:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11768.821586] Lustre: Skipped 1 previous similar message [11769.884184] Lustre: server umount lustre-MDT0001 complete [11774.503888] Lustre: server umount lustre-OST0000 complete [11784.079812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11789.335875] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11792.003702] Lustre: 336072:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.134@tcp [11794.127210] Lustre: server umount MGS complete [11801.273899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11801.712060] LustreError: 336645:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11801.768802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11806.119608] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11814.798557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11815.158274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11819.871902] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11829.921713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11830.656633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11837.377742] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11844.231689] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11845.799076] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11851.304244] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11852.922435] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11857.380328] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11857.386766] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11857.395252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11857.412208] Lustre: Skipped 1 previous similar message [11857.426265] Lustre: Skipped 1 previous similar message [11859.556587] Lustre: server umount lustre-OST0000 complete [11866.604434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11866.621072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11866.635474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11870.153770] Lustre: server umount lustre-MDT0000 complete [11872.740681] LustreError: 336646:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11872.768874] LustreError: 336646:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11874.098541] LustreError: 336625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789460105 with bad export cookie 5088916396791691348 [11874.111738] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11874.114276] LustreError: 336625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11874.535112] Lustre: server umount lustre-MDT0001 complete [11885.446534] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [11888.393396] Key type lgssc unregistered [11888.740620] LNet: 340079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11888.748861] LNetError: 340079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11888.778272] LNet: Removed LNI 192.168.204.134@tcp [11889.888468] Key type .llcrypt unregistered [11889.892515] Key type ._llcrypt unregistered [11908.653461] Key type ._llcrypt registered [11908.655741] Key type .llcrypt registered [11908.812578] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [11927.824806] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [11928.886833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11928.922789] alg: No test for adler32 (adler32-zlib) [11930.113397] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [11930.461496] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [11932.151196] Key type lgssc registered [11933.559698] Lustre: Echo OBD driver; http://www.lustre.org/ [11972.103109] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11972.123817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11973.446657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11973.481160] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11973.599414] Lustre: lustre-MDT0000: new disk, initializing [11973.727144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11973.766708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11979.803135] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [11991.408773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11991.586381] Lustre: 343238:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11991.694361] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11991.700133] Lustre: Skipped 1 previous similar message [11991.777928] Lustre: lustre-MDT0001: new disk, initializing [11991.887223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11991.935507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11991.944171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11997.722194] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [12008.862026] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12015.335610] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12023.975282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12024.443218] Lustre: lustre-OST0000: new disk, initializing [12024.449061] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12024.459156] Lustre: 345205:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12024.561841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12031.092984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12031.103195] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12031.234686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12032.240699] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [12043.825434] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12049.629710] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12049.820620] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12055.270171] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12055.531272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12062.179034] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12062.194983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12062.224951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12063.710808] Lustre: server umount lustre-OST0000 complete [12068.832514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12068.842784] LustreError: Skipped 1 previous similar message [12068.853053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12068.876780] Lustre: Skipped 1 previous similar message [12068.887413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12068.890673] Lustre: Skipped 1 previous similar message [12072.417706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12072.429359] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12073.899360] Lustre: server umount lustre-MDT0000 complete [12078.213364] LustreError: 343229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789460309 with bad export cookie 16304837877936946932 [12078.215580] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12078.224211] LustreError: 343229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12078.512570] Lustre: server umount lustre-MDT0001 complete [12090.201133] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 04:18:38 (1789460318) [12106.216232] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [12120.527933] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [12162.638871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12163.023640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12163.056608] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12163.130646] Lustre: lustre-MDT0000: new disk, initializing [12163.226547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12163.249082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12168.472108] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [12177.894555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12178.008931] Lustre: 349967:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12178.060195] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12178.064473] Lustre: Skipped 1 previous similar message [12178.142400] Lustre: lustre-MDT0001: new disk, initializing [12178.263777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12178.295246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12178.316683] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12184.451527] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [12195.340644] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12201.166623] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12208.248191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12208.481203] Lustre: lustre-OST0000: new disk, initializing [12208.484329] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12208.490706] Lustre: 351934:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12208.568618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12215.242583] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [12215.883983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12215.907439] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12216.098823] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12224.255280] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12257.253871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12257.254270] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12257.265135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12257.265144] Lustre: Skipped 1 previous similar message [12257.270581] Lustre: Skipped 1 previous similar message [12257.322738] LustreError: Skipped 1 previous similar message [12262.055870] Lustre: server umount lustre-OST0000 complete [12270.561575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12270.567704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12270.598869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12270.609814] Lustre: Skipped 1 previous similar message [12275.495554] Lustre: server umount lustre-MDT0000 complete [12276.706184] LustreError: 349979:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12276.745259] LustreError: 349979:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12281.231799] LustreError: 349958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789460512 with bad export cookie 16304837877936947807 [12281.239021] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12281.247115] LustreError: 349958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12281.984433] Lustre: server umount lustre-MDT0001 complete [12295.290130] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [12298.810828] Key type lgssc unregistered [12299.064672] LNet: 354090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12299.077437] LNetError: 354090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12299.105870] LNet: Removed LNI 192.168.204.134@tcp [12300.225175] Key type .llcrypt unregistered [12300.228374] Key type ._llcrypt unregistered [12400.290126] Key type ._llcrypt registered [12400.292409] Key type .llcrypt registered [12400.479766] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [12401.038318] Key type .llcrypt unregistered [12401.044230] Key type ._llcrypt unregistered [12474.978251] Key type ._llcrypt registered [12474.985405] Key type .llcrypt registered [12475.115612] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [12475.453333] Key type .llcrypt unregistered [12475.456276] Key type ._llcrypt unregistered [12504.902226] Key type ._llcrypt registered [12504.911822] Key type .llcrypt registered [12505.153376] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [12505.769300] Key type .llcrypt unregistered [12505.787521] Key type ._llcrypt unregistered [12540.425838] Key type ._llcrypt registered [12540.427391] Key type .llcrypt registered [12540.565341] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [12540.862834] Key type .llcrypt unregistered [12540.866095] Key type ._llcrypt unregistered [12611.244338] Key type ._llcrypt registered [12611.246185] Key type .llcrypt registered [12611.342462] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [12611.609377] Key type .llcrypt unregistered [12611.611890] Key type ._llcrypt unregistered [12641.980332] Key type ._llcrypt registered [12641.982869] Key type .llcrypt registered [12642.071676] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [12642.366423] Key type .llcrypt unregistered [12642.369232] Key type ._llcrypt unregistered [12672.499595] Key type ._llcrypt registered [12672.505509] Key type .llcrypt registered [12672.612718] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [12672.917119] Key type .llcrypt unregistered [12672.918776] Key type ._llcrypt unregistered [12698.760721] Key type ._llcrypt registered [12698.764528] Key type .llcrypt registered [12698.870773] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [12699.171467] Key type .llcrypt unregistered [12699.177672] Key type ._llcrypt unregistered [12730.776482] Key type ._llcrypt registered [12730.779209] Key type .llcrypt registered [12730.881890] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [12731.149348] Key type .llcrypt unregistered [12731.151823] Key type ._llcrypt unregistered [12762.196880] Key type ._llcrypt registered [12762.200899] Key type .llcrypt registered [12762.324192] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [12762.631306] Key type .llcrypt unregistered [12762.635610] Key type ._llcrypt unregistered [12822.632173] Key type ._llcrypt registered [12822.634694] Key type .llcrypt registered [12822.763325] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [12823.050324] Key type .llcrypt unregistered [12823.057383] Key type ._llcrypt unregistered [12854.655847] Key type ._llcrypt registered [12854.658348] Key type .llcrypt registered [12854.771761] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [12855.030365] Key type .llcrypt unregistered [12855.033386] Key type ._llcrypt unregistered [12883.460173] Key type ._llcrypt registered [12883.463242] Key type .llcrypt registered [12883.547680] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [12883.793774] Key type .llcrypt unregistered [12883.798945] Key type ._llcrypt unregistered [12934.438847] Key type ._llcrypt registered [12934.441226] Key type .llcrypt registered [12934.571776] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [12949.053923] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [12949.867591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12949.903618] alg: No test for adler32 (adler32-zlib) [12950.963831] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [12951.219390] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [12952.879264] Key type lgssc registered [12954.070721] Lustre: Echo OBD driver; http://www.lustre.org/ [12985.614166] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12985.647378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12986.837313] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12986.873769] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12986.978238] Lustre: tst250fs-MDT0000: new disk, initializing [12987.039636] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12987.060139] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12991.177748] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [13001.156509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13001.273580] Lustre: 374338:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13001.335529] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13001.339959] Lustre: Skipped 1 previous similar message [13001.360181] Lustre: tst250fs-MDT0001: Not available for connect from 0@lo (not set up) [13001.419376] Lustre: tst250fs-MDT0001: new disk, initializing [13001.511853] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13001.552528] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13001.568082] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13005.645810] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [13010.577911] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13016.974824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13017.278801] Lustre: tst250fs-OST0000: new disk, initializing [13017.282940] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13017.293492] Lustre: 375972:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13017.371827] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13017.991138] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13018.002895] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13018.124735] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13023.824375] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [13034.268282] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13034.463554] Lustre: tst250fs-OST0001: new disk, initializing [13034.467181] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13034.472204] Lustre: 376846:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13034.565368] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13041.194105] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13041.201525] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13041.346563] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13041.943997] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [13048.089114] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13050.651352] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13050.662208] Lustre: Skipped 1 previous similar message [13072.353594] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13072.365697] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13072.372680] Lustre: Skipped 2 previous similar messages [13072.396955] Lustre: Skipped 3 previous similar messages [13076.017465] Lustre: server umount tst250fs-MDT0000 complete [13079.118638] LustreError: 374330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789461310 with bad export cookie 5741510353021034821 [13079.123252] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13079.125587] LustreError: 374330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13079.419677] Lustre: server umount tst250fs-MDT0001 complete [13093.494966] Lustre: server umount tst250fs-OST0000 complete [13107.158332] Lustre: server umount tst250fs-OST0001 complete [13124.082389] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [13131.503672] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [13158.186236] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 04:36:27 (1789461387) [13159.749484] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13172.365343] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [13182.538877] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [13209.299975] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13076 sec ========= 04:37:18 (1789461438) [13210.794961] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:37:19 (1789461439) === [13213.593873] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:37:22 (1789461442) === [13245.070482] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [13247.415417] Key type lgssc unregistered [13247.670015] LNet: 385967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13247.680501] LNetError: 385967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13247.691115] LNet: Removed LNI 192.168.204.134@tcp [13248.393312] Key type .llcrypt unregistered [13248.396619] Key type ._llcrypt unregistered