[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 511899396 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003210] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008318] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.012079] LSM: Security Framework initializing [ 0.013053] Yama: becoming mindful. [ 0.014038] SELinux: Initializing. [ 0.015069] *** VALIDATE selinux *** [ 0.023801] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028253] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030107] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031115] *** VALIDATE tmpfs *** [ 0.033065] *** VALIDATE proc *** [ 0.034261] *** VALIDATE cgroup *** [ 0.035012] *** VALIDATE cgroup2 *** [ 0.036288] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038095] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040029] Spectre V2 : User space: Vulnerable [ 0.041008] Speculative Store Bypass: Vulnerable [ 0.044450] debug: unmapping init [mem 0xffffffffbc859000-0xffffffffbc860fff] [ 0.046170] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047675] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048025] ... version: 2 [ 0.049012] ... bit width: 48 [ 0.050014] ... generic registers: 4 [ 0.051013] ... value mask: 0000ffffffffffff [ 0.052015] ... max period: 00007fffffffffff [ 0.053015] ... fixed-purpose events: 3 [ 0.054012] ... event mask: 000000070000000f [ 0.055288] rcu: Hierarchical SRCU implementation. [ 0.057423] smp: Bringing up secondary CPUs ... [ 0.058547] x86: Booting SMP configuration: [ 0.059022] .... node #0, CPUs: #1 #2 #3 [ 0.062310] smp: Brought up 1 node, 4 CPUs [ 0.064013] smpboot: Max logical packages: 1 [ 0.065024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.140019] node 0 deferred pages initialised in 73ms [ 0.143118] devtmpfs: initialized [ 0.144299] x86/mm: Memory block size: 128MB [ 0.147091] gcov: version magic: 0x41383552 [ 0.149359] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151277] pinctrl core: initialized pinctrl subsystem [ 0.152231] [ 0.152798] ************************************************************* [ 0.153014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154015] ** ** [ 0.155018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156013] ** ** [ 0.157013] ** This means that this kernel is built to expose internal ** [ 0.158014] ** IOMMU data structures, which may compromise security on ** [ 0.159013] ** your system. ** [ 0.160013] ** ** [ 0.161014] ** If you see this message and you are not debugging the ** [ 0.162012] ** kernel, report this immediately to your vendor! ** [ 0.163014] ** ** [ 0.164012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165013] ************************************************************* [ 0.166720] NET: Registered protocol family 16 [ 0.167517] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.168060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.169064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.170439] cpuidle: using governor menu [ 0.173438] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.178761] PCI: Using configuration type 1 for base access [ 0.184131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.198191] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.199096] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.201190] cryptd: max_cpu_qlen set to 1000 [ 0.203269] ACPI: Added _OSI(Module Device) [ 0.204012] ACPI: Added _OSI(Processor Device) [ 0.205012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.206013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.210159] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.212607] ACPI: Interpreter enabled [ 0.213061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.214016] ACPI: Using IOAPIC for interrupt routing [ 0.215121] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.216352] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.225940] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.226039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.227018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.228072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.230280] acpiphp: Slot [2] registered [ 0.231132] acpiphp: Slot [5] registered [ 0.232106] acpiphp: Slot [6] registered [ 0.233115] acpiphp: Slot [7] registered [ 0.234116] acpiphp: Slot [8] registered [ 0.235105] acpiphp: Slot [9] registered [ 0.236146] acpiphp: Slot [10] registered [ 0.237097] acpiphp: Slot [3] registered [ 0.238095] acpiphp: Slot [4] registered [ 0.239114] acpiphp: Slot [11] registered [ 0.240103] acpiphp: Slot [12] registered [ 0.241108] acpiphp: Slot [13] registered [ 0.242106] acpiphp: Slot [14] registered [ 0.243093] acpiphp: Slot [15] registered [ 0.244116] acpiphp: Slot [16] registered [ 0.245098] acpiphp: Slot [17] registered [ 0.246094] acpiphp: Slot [18] registered [ 0.247092] acpiphp: Slot [19] registered [ 0.248061] acpiphp: Slot [20] registered [ 0.249080] acpiphp: Slot [21] registered [ 0.250103] acpiphp: Slot [22] registered [ 0.251111] acpiphp: Slot [23] registered [ 0.252143] acpiphp: Slot [24] registered [ 0.253105] acpiphp: Slot [25] registered [ 0.254179] acpiphp: Slot [26] registered [ 0.255100] acpiphp: Slot [27] registered [ 0.256101] acpiphp: Slot [28] registered [ 0.257083] acpiphp: Slot [29] registered [ 0.258000] acpiphp: Slot [30] registered [ 0.260116] acpiphp: Slot [31] registered [ 0.261085] PCI host bridge to bus 0000:00 [ 0.263021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.266024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.269022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.273049] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.276032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.279047] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.281262] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.283791] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.287214] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.297832] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.302054] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.305024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.307017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.313018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.316306] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.319768] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.322042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.326953] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.333015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.349016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.355018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.362047] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.371038] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.382018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.409017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.426546] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.435022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.442026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.462024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.472000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.482016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.494013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.517015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.526000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.531022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.544023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.570023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.580629] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.589019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.604020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.630023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.643277] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.650023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.656020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.674023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.689751] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.692340] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.695405] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.697456] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.700421] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.704110] iommu: Default domain type: Passthrough [ 0.707598] SCSI subsystem initialized [ 0.709183] ACPI: bus type USB registered [ 0.711117] usbcore: registered new interface driver usbfs [ 0.713103] usbcore: registered new interface driver hub [ 0.715112] usbcore: registered new device driver usb [ 0.717273] pps_core: LinuxPPS API ver. 1 registered [ 0.719020] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.723085] PTP clock support registered [ 0.725109] EDAC MC: Ver: 3.0.0 [ 0.727513] PCI: Using ACPI for IRQ routing [ 0.730034] NetLabel: Initializing [ 0.731013] NetLabel: domain hash size = 128 [ 0.733020] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.735089] NetLabel: unlabeled traffic allowed by default [ 0.738076] vgaarb: loaded [ 0.739663] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.741014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.748328] clocksource: Switched to clocksource kvm-clock [ 0.860623] VFS: Disk quotas dquot_6.6.0 [ 0.861968] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.863889] *** VALIDATE ramfs *** [ 0.864954] *** VALIDATE hugetlbfs *** [ 0.866725] pnp: PnP ACPI init [ 0.869439] pnp: PnP ACPI: found 6 devices [ 0.885097] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.888398] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.890908] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.892790] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.895218] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.897798] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.900492] NET: Registered protocol family 2 [ 0.903107] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.907505] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.910862] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.915849] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.919483] TCP: Hash tables configured (established 65536 bind 65536) [ 0.921976] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.924864] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.927612] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.930481] NET: Registered protocol family 1 [ 0.933364] RPC: Registered named UNIX socket transport module. [ 0.937695] RPC: Registered udp transport module. [ 0.940891] RPC: Registered tcp transport module. [ 0.942923] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.946375] NET: Registered protocol family 44 [ 0.948835] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.951249] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.953836] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.956951] PCI: CLS 0 bytes, default 64 [ 0.959083] Unpacking initramfs... [ 2.405567] debug: unmapping init [mem 0xffff9ff83cc54000-0xffff9ff83ffbffff] [ 2.409471] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.411116] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.413524] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.960105] Initialise system trusted keyrings [ 2.961932] Key type blacklist registered [ 2.963821] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.972754] zbud: loaded [ 2.975389] *** VALIDATE nfs *** [ 2.976843] *** VALIDATE nfs4 *** [ 2.978612] pstore: using deflate compression [ 2.982111] Platform Keyring initialized [ 3.126975] NET: Registered protocol family 38 [ 3.129067] Key type asymmetric registered [ 3.130587] Asymmetric key parser 'x509' registered [ 3.132202] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.134931] io scheduler mq-deadline registered [ 3.136451] io scheduler kyber registered [ 3.138073] io scheduler bfq registered [ 3.140254] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.143968] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.148426] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.152224] ACPI: Power Button [PWRF] [ 3.159803] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.167976] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.187977] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.199554] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.229714] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.259203] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.288618] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.295053] Non-volatile memory driver v1.3 [ 3.297000] Linux agpgart interface v0.103 [ 3.332303] virtio_blk virtio1: [vda] 146152 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.336768] vda: detected capacity change from 0 to 74829824 [ 3.354827] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.357845] vdb: detected capacity change from 0 to 1073741824 [ 3.376481] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.380191] vdc: detected capacity change from 0 to 2621440000 [ 3.395594] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.398431] vdd: detected capacity change from 0 to 2621440000 [ 3.414680] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.417614] vde: detected capacity change from 0 to 4294967296 [ 3.440672] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.445193] vdf: detected capacity change from 0 to 4294967296 [ 3.457267] libphy: Fixed MDIO Bus: probed [ 3.464744] usbcore: registered new interface driver usbserial_generic [ 3.467431] usbserial: USB Serial support registered for generic [ 3.469889] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.475477] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.477331] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.479732] mousedev: PS/2 mouse device common for all mice [ 3.482694] rtc_cmos 00:05: RTC can wake from S4 [ 3.485284] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.489809] rtc_cmos 00:05: registered as rtc0 [ 3.491593] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.494370] intel_pstate: CPU model not supported [ 3.498032] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.499456] hid: raw HID events driver (C) Jiri Kosina [ 3.506718] usbcore: registered new interface driver usbhid [ 3.508452] usbhid: USB HID core driver [ 3.509907] drop_monitor: Initializing network drop monitor service [ 3.512351] Initializing XFRM netlink socket [ 3.512416] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.514050] NET: Registered protocol family 10 [ 3.519148] Segment Routing with IPv6 [ 3.520382] NET: Registered protocol family 17 [ 3.522573] mpls_gso: MPLS GSO support [ 3.528886] RAS: Correctable Errors collector initialized. [ 3.531084] AVX version of gcm_enc/dec engaged. [ 3.533149] AES CTR mode by8 optimization enabled [ 3.613572] sched_clock: Marking stable (3613490181, 0)->(4583933305, -970443124) [ 3.617968] registered taskstats version 1 [ 3.621226] Loading compiled-in X.509 certificates [ 3.623229] zswap: loaded using pool lzo/zbud [ 3.657551] Key type big_key registered [ 3.673106] Key type encrypted registered [ 3.675357] ima: No TPM chip found, activating TPM-bypass! [ 3.677525] ima: Allocated hash algorithm: sha1 [ 3.679870] ima: No architecture policies found [ 3.683125] evm: Initialising EVM extended attributes: [ 3.685199] evm: security.selinux [ 3.686448] evm: security.ima [ 3.687955] evm: security.capability [ 3.689959] evm: HMAC attrs: 0x1 [ 3.693312] rtc_cmos 00:05: setting system clock to 2026-08-24 19:14:47 UTC (1787598887) [ 3.700987] debug: unmapping init [mem 0xffffffffbd803000-0xffffffffbd9fffff] [ 3.704339] debug: unmapping init [mem 0xffffffffbc582000-0xffffffffbc858fff] [ 3.713503] Write protecting the kernel read-only data: 28672k [ 3.716913] debug: unmapping init [mem 0xffffffffbac03000-0xffffffffbadfffff] [ 3.720130] debug: unmapping init [mem 0xffffffffbb514000-0xffffffffbb5fffff] [ 3.759052] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.768914] systemd[1]: Detected virtualization kvm. [ 3.770698] systemd[1]: Detected architecture x86-64. [ 3.772542] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.797912] systemd[1]: No hostname configured. [ 3.799892] systemd[1]: Set hostname to . [ 3.802427] random: systemd: uninitialized urandom read (16 bytes read) [ 3.805505] systemd[1]: Initializing machine ID from random generator. [ 3.856069] random: ln: uninitialized urandom read (6 bytes read) [ 3.971332] random: systemd: uninitialized urandom read (16 bytes read) [ 3.974177] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.979942] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.993485] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.643178] device-mapper: uevent: version 1.0.3 [ 4.645996] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK [ 5.323905] random: fast init done ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.469482] virtio_net virtio0 ens2: renamed from eth0 [ 5.581625] scsi host0: ata_piix [ 5.614735] scsi host1: ata_piix [ 5.621377] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.623823] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.126801] hrtimer: interrupt took 3229680 ns [ 10.150335] dracut-initqueue[594]: RTNETLINK answers: File exists [ 10.196523] random: crng init done [ 10.197529] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.871116] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.311527] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.625768] SELinux: Disabled at runtime. [ 12.688590] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.698582] systemd[1]: Detected virtualization kvm. [ 12.700265] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.326290] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.330753] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.336497] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.340948] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.346996] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.367622] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.380677] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. [ 13.658052] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.979256] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.431232] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.456872] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.626248] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.684328] EDAC sbridge: Ver: 1.1.2 [ 16.613696] Key type dns_resolver registered [ 16.956977] NFS: Registering the id_resolver key type [ 16.962148] Key type id_resolver registered [ 16.963868] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg603-server login: [ 61.634329] libcfs: loading out-of-tree module taints kernel. [ 61.672629] Key type ._llcrypt registered [ 61.677802] Key type .llcrypt registered [ 61.844662] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 81.022553] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 82.791742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 82.830015] alg: No test for adler32 (adler32-zlib) [ 84.488135] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 85.669720] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 87.511945] Key type lgssc registered [ 89.774557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 106.597200] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 156.274206] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 170.601199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 170.627451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 171.970980] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 172.054331] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 172.236327] Lustre: lustre-MDT0000: new disk, initializing [ 172.339353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 172.377123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 177.558749] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.289300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 190.358326] Lustre: 6516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 190.383647] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 190.386488] Lustre: Skipped 1 previous similar message [ 190.437498] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 190.448579] Lustre: lustre-MDT0001: new disk, initializing [ 190.542204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 190.591046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 190.608083] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 195.297453] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.352135] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 210.710526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 211.048173] Lustre: lustre-OST0000: new disk, initializing [ 211.051618] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 211.056938] Lustre: 8453:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 211.117448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 212.094654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 212.109250] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 212.194974] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 217.072747] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.549890] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 231.700899] Lustre: lustre-OST0001: new disk, initializing [ 231.707761] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 231.716780] Lustre: 9526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 231.793037] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 238.872552] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.743815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 240.761596] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 240.827158] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 253.102809] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 262.326550] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 268.670858] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing check_logdir /tmp/testlogs/ [ 273.109525] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing yml_node [ 277.806642] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 280.613627] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 283.572260] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 285.259140] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 24 15:19:27 EDT 2026 [ 303.414495] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 304.856503] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 323.041281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 323.043953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 323.068363] Lustre: Skipped 3 previous similar messages [ 323.098107] Lustre: Skipped 3 previous similar messages [ 328.161790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 328.175877] Lustre: Skipped 3 previous similar messages [ 333.281190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 333.293789] Lustre: Skipped 3 previous similar messages [ 335.328281] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 335.615233] Lustre: server umount lustre-MDT0000 complete [ 339.533905] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787599223 with bad export cookie 9927404549382310999 [ 339.543062] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 343.522118] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.522179] LustreError: 6524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.531056] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 343.534890] Lustre: Skipped 1 previous similar message [ 343.564899] LustreError: 6524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 344.546231] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 344.555892] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 345.822841] Lustre: server umount lustre-MDT0001 complete [ 350.033334] Lustre: server umount lustre-OST0000 complete [ 355.031654] Lustre: server umount lustre-OST0001 complete [ 362.652201] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 373.192538] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 410.716082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 411.017323] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 411.123338] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 411.186712] Lustre: lustre-MDT0000: new disk, initializing [ 411.295832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 411.328060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 416.257808] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 427.524615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 427.657815] Lustre: 15082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 427.719456] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 427.722384] Lustre: Skipped 1 previous similar message [ 427.805833] Lustre: lustre-MDT0001: new disk, initializing [ 427.901524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 427.941099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 427.951987] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 432.656493] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 441.877221] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 447.761469] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 455.337182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 455.714312] Lustre: lustre-OST0000: new disk, initializing [ 455.718619] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 455.726080] Lustre: 17049:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 455.823719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 457.142068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 457.157162] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 457.277094] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 462.337416] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 471.012824] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 475.349266] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 475.650695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 480.285614] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 480.516654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 482.795404] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 482.797800] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 482.805745] LustreError: Skipped 1 previous similar message [ 482.808548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 482.821924] Lustre: Skipped 1 previous similar message [ 482.843254] Lustre: Skipped 4 previous similar messages [ 488.419926] Lustre: server umount lustre-OST0000 complete [ 493.030265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 493.063036] Lustre: Skipped 1 previous similar message [ 498.654893] Lustre: server umount lustre-MDT0000 complete [ 502.777329] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787599386 with bad export cookie 9927404549382312532 [ 502.782229] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 502.795880] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 503.076960] Lustre: server umount lustre-MDT0001 complete [ 508.516719] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 15:23:10 (1787599390) [ 525.309903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 525.820385] LustreError: 19924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.899966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 530.255489] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 530.923766] LustreError: 19925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.040241] LustreError: 19924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 539.141711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 544.123477] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 552.062924] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 557.520984] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 559.566029] Lustre: Failing over lustre-MDT0000 [ 559.858610] Lustre: server umount lustre-MDT0000 complete [ 560.096461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 560.107217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 560.134867] LustreError: 19924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.671649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 578.816549] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 583.119201] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 584.161904] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 584.171369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 584.195530] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 592.411319] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 596.215436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 596.231604] Lustre: Skipped 2 previous similar messages [ 596.240318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 596.248557] Lustre: Skipped 7 previous similar messages [ 602.576575] Lustre: server umount lustre-MDT0000 complete [ 606.263180] LustreError: 20659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787599490 with bad export cookie 9927404549382313946 [ 606.266748] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 606.276245] LustreError: 20659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 606.652032] Lustre: server umount lustre-MDT0001 complete [ 616.398990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 616.963908] LustreError: 23261:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 616.984429] LustreError: 23261:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 617.046303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 617.054636] Lustre: Skipped 2 previous similar messages [ 622.084687] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 630.444415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 635.419750] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 643.360945] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 649.230790] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 651.275708] Lustre: Failing over lustre-MDT0000 [ 651.490566] Lustre: server umount lustre-MDT0000 complete [ 651.744537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 651.746480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 651.752111] LustreError: 23989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 651.763172] Lustre: Skipped 1 previous similar message [ 651.780525] LustreError: 23989:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 671.199563] Lustre: 3647:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787599539/real 1787599539] req@ffff9ff7832bd880 x1874433376709632/t0(0) o400->MGC192.168.206.103@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787599555 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 671.223616] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 671.586539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 682.762426] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 687.159717] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 688.102908] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 688.107956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 688.129787] Lustre: Skipped 1 previous similar message [ 688.166798] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 697.456944] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 703.457537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 703.474160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 703.499987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 703.509170] Lustre: Skipped 2 previous similar messages [ 707.764508] Lustre: server umount lustre-MDT0000 complete [ 711.617331] LustreError: 23242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787599595 with bad export cookie 9927404549382314884 [ 711.627311] LustreError: 23242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 711.627509] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 712.015643] Lustre: server umount lustre-MDT0001 complete [ 721.860184] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 15:26:43 (1787599603) [ 739.389567] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 749.759843] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 790.246733] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 801.945881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 802.296546] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 802.342892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 802.437566] Lustre: lustre-MDT0000: new disk, initializing [ 802.501792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 802.507383] Lustre: Skipped 2 previous similar messages [ 802.515964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 806.804928] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 816.830643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 816.981791] Lustre: 29637:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 817.259302] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 821.406933] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 830.055880] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 834.962761] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 841.902011] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 842.264834] Lustre: 31605:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 843.643908] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 843.687640] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 843.761465] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 843.782234] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 848.974778] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 858.922821] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 864.774345] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 884.709034] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 884.715688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 884.719240] Lustre: Skipped 1 previous similar message [ 884.721884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 884.740025] Lustre: Skipped 3 previous similar messages [ 889.730759] Lustre: server umount lustre-OST0000 complete [ 901.125597] Lustre: server umount lustre-MDT0000 complete [ 904.225411] LustreError: 29648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 904.259764] LustreError: 29648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [ 907.130914] LustreError: 30431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787599790 with bad export cookie 9927404549382315290 [ 907.139155] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 907.149507] LustreError: 30431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 907.822580] Lustre: server umount lustre-MDT0001 complete [ 921.829368] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 924.346346] Key type lgssc unregistered [ 924.645070] LNet: 33591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.659859] LNetError: 33591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 924.680170] LNet: Removed LNI 192.168.206.103@tcp [ 925.798379] Key type .llcrypt unregistered [ 925.800554] Key type ._llcrypt unregistered [ 948.182898] Key type ._llcrypt registered [ 948.187477] Key type .llcrypt registered [ 948.344093] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 949.556644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 949.590882] alg: No test for adler32 (adler32-zlib) [ 950.802139] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 951.041502] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 952.704403] Key type lgssc registered [ 953.937380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 961.344416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 961.357717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 962.716716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 965.592436] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 972.260281] LustreError: 35010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 972.275031] LustreError: 35010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 972.962188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 973.274914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 977.922283] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 986.192653] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 992.728104] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 999.587216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 999.985635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1005.827049] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1008.216785] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1013.010667] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1018.426924] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1030.338598] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 15:31:52 (1787599912) [ 1032.200628] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1034.110089] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1041.102205] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 15:32:03 (1787599923) [ 1049.173067] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 15:32:11 (1787599931) [ 1050.163893] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1058.256536] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1058.263310] Lustre: Skipped 2 previous similar messages [ 1059.884750] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 15:32:22 (1787599942) [ 1063.901022] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1063.911725] Lustre: Skipped 3 previous similar messages [ 1081.587121] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1081.594422] Lustre: Skipped 6 previous similar messages [ 1084.635098] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 15:32:46 (1787599966) [ 1087.626422] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1091.180857] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1100.444213] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 15:33:02 (1787599982) [ 1102.727694] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1102.736496] Lustre: Skipped 3 previous similar messages [ 1116.565628] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 15:33:18 (1787599998) [ 1136.097589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1136.098216] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1136.109018] LustreError: Skipped 1 previous similar message [ 1136.112422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1136.127035] Lustre: Skipped 1 previous similar message [ 1141.216887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1141.229311] Lustre: Skipped 2 previous similar messages [ 1146.339697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1146.343763] Lustre: Skipped 1 previous similar message [ 1147.359152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1147.551375] Lustre: server umount lustre-OST0000 complete [ 1151.456206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1151.457026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1151.466973] Lustre: Skipped 1 previous similar message [ 1151.478714] Lustre: Skipped 1 previous similar message [ 1156.578640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1156.585557] Lustre: Skipped 1 previous similar message [ 1157.584911] Lustre: server umount lustre-MDT0000 complete [ 1161.050016] LustreError: 34990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787600044 with bad export cookie 5693869680303685406 [ 1161.053303] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1161.069810] LustreError: 34990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1161.341359] Lustre: server umount lustre-MDT0001 complete [ 1172.249278] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 1175.091773] Key type lgssc unregistered [ 1175.410981] LNet: 42875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1175.427409] LNetError: 42875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1175.453111] LNet: Removed LNI 192.168.206.103@tcp [ 1176.378538] Key type .llcrypt unregistered [ 1176.383191] Key type ._llcrypt unregistered [ 1194.934271] Key type ._llcrypt registered [ 1194.938287] Key type .llcrypt registered [ 1195.076384] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 1210.973478] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 1212.310584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1212.365246] alg: No test for adler32 (adler32-zlib) [ 1213.441033] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 1213.640767] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 1215.359188] Key type lgssc registered [ 1216.603158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1250.211743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1250.225132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1251.532068] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1251.588181] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1251.689633] Lustre: lustre-MDT0000: new disk, initializing [ 1251.781195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1251.796234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1255.774840] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1266.794478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1266.882778] Lustre: 46028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1266.933277] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1266.937449] Lustre: Skipped 1 previous similar message [ 1266.995668] Lustre: lustre-MDT0001: new disk, initializing [ 1267.118597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1267.160508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1267.169305] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1271.713173] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1282.162412] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1287.765489] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1293.878306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1294.205981] Lustre: lustre-OST0000: new disk, initializing [ 1294.209587] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1294.220726] Lustre: 47994:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1294.323328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1300.623816] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1304.109263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1304.118608] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1304.216362] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1310.831620] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1315.423270] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1315.718330] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1320.501689] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1320.701744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1325.028744] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1325.030758] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1325.037116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1325.043280] Lustre: Skipped 1 previous similar message [ 1325.056829] Lustre: Skipped 1 previous similar message [ 1328.935862] Lustre: server umount lustre-OST0000 complete [ 1333.743982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1333.751291] LustreError: Skipped 1 previous similar message [ 1333.759314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1333.792732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1335.265916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1335.267914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1335.312176] Lustre: Skipped 1 previous similar message [ 1339.005524] Lustre: server umount lustre-MDT0000 complete [ 1342.846513] LustreError: 47993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787600226 with bad export cookie 14889909106697695099 [ 1342.852860] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1342.860239] LustreError: 47993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1343.443174] Lustre: server umount lustre-MDT0001 complete [ 1352.830745] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 15:37:15 (1787600235) [ 1359.021260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1359.423334] LustreError: 50457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1359.444460] LustreError: 50457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1359.491536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1363.517877] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1364.980795] LustreError: 50458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1370.095625] LustreError: 50457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1372.452123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1372.841185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1377.286760] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1385.686389] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1391.799631] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1397.987672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1398.318986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1404.575622] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1411.506234] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1419.314506] LustreError: 52987:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1420.251546] Lustre: Modifying parameter general.timeout=1 in log params [ 1422.023892] Lustre: Modifying parameter general.timeout=3 in log params [ 1422.026178] Lustre: Skipped 1 previous similar message [ 1424.689092] Lustre: Modifying parameter general.timeout=6 in log params [ 1424.691373] Lustre: Skipped 2 previous similar messages [ 1429.123295] Lustre: Modifying parameter general.timeout=11 in log params [ 1429.125730] Lustre: Skipped 4 previous similar messages [ 1437.962958] Lustre: Modifying parameter general.timeout=21 in log params [ 1437.969645] Lustre: Skipped 9 previous similar messages [ 1454.034372] Lustre: Modifying parameter general.timeout=37 in log params [ 1454.045632] Lustre: Skipped 15 previous similar messages [ 1486.549750] Lustre: Modifying parameter general.timeout=74 in log params [ 1486.555733] Lustre: Skipped 36 previous similar messages [ 1540.134218] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 15:40:22 (1787600422) [ 1543.091640] Lustre: 59924:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1544.867214] Lustre: 60022:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1548.367568] Lustre: 60236:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1548.380500] Lustre: 60236:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1550.746337] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 in log params [ 1550.749788] Lustre: Skipped 62 previous similar messages [ 1576.172849] Lustre: 61795:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1576.183634] Lustre: 61795:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1576.219328] LustreError: 61795:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1580.333598] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 15:41:02 (1787600462) [ 1592.801080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1592.807838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1592.818540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1597.527586] Lustre: server umount lustre-MDT0000 complete [ 1600.485630] LustreError: 50457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1600.500627] LustreError: 50457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1600.826787] LustreError: 50439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787600484 with bad export cookie 14889909106697695981 [ 1600.830983] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1600.839415] LustreError: 50439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1601.041858] Lustre: server umount lustre-MDT0001 complete [ 1615.250993] Lustre: server umount lustre-OST0000 complete [ 1643.750509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1644.157238] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1644.442911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1649.539696] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1659.060871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1659.211816] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1659.241178] Lustre: 64326:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1664.861613] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1672.687324] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1678.573961] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1686.256090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1686.416087] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1686.673238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1686.683294] Lustre: Skipped 1 previous similar message [ 1693.829741] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1701.991909] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1703.940762] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1703.947868] Lustre: Skipped 16 previous similar messages [ 1704.030476] Lustre: 66893:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1712.608273] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1712.615446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1712.634214] Lustre: Skipped 2 previous similar messages [ 1712.643043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1712.648903] Lustre: Skipped 3 previous similar messages [ 1718.192255] Lustre: server umount lustre-OST0000 complete [ 1723.360793] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1723.399657] Lustre: Skipped 1 previous similar message [ 1723.412226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1723.440052] Lustre: Skipped 3 previous similar messages [ 1728.388841] Lustre: server umount lustre-MDT0000 complete [ 1728.482290] LustreError: 65833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1728.497904] LustreError: 65833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1732.595806] LustreError: 65067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787600616 with bad export cookie 14889909106697698984 [ 1732.596271] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1732.605295] LustreError: 65067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1733.107673] Lustre: server umount lustre-MDT0001 complete [ 1745.478215] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 1748.551389] Key type lgssc unregistered [ 1748.874417] LNet: 68107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1748.884287] LNetError: 68107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1748.905691] LNet: Removed LNI 192.168.206.103@tcp [ 1749.832654] Key type .llcrypt unregistered [ 1749.841254] Key type ._llcrypt unregistered [ 1777.346283] Key type ._llcrypt registered [ 1777.348397] Key type .llcrypt registered [ 1777.449599] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 1778.267442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1778.338544] alg: No test for adler32 (adler32-zlib) [ 1779.388588] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 1779.607898] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 1781.280249] Key type lgssc registered [ 1782.466321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1789.879951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1789.906612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1791.342296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1796.396971] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1801.701433] LustreError: 69519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1801.720334] LustreError: 69519:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1804.604408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1805.215662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1809.342191] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1816.616397] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1822.190971] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1828.731868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1829.172928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1834.778569] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1842.471461] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1849.212266] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1856.994575] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1857.010695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1857.014865] Lustre: Skipped 1 previous similar message [ 1861.600189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1861.606593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1861.618720] Lustre: Skipped 1 previous similar message [ 1861.625441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1863.175266] Lustre: server umount lustre-MDT0000 complete [ 1866.933286] LustreError: 69499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787600750 with bad export cookie 13029069235374584187 [ 1866.946304] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1866.949288] LustreError: 69499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1867.221636] Lustre: server umount lustre-MDT0001 complete [ 1880.522977] Lustre: server umount lustre-OST0000 complete [ 1909.457640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1909.829771] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1909.868288] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1910.183316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1915.204603] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1923.731750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1923.845291] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1923.872812] Lustre: 74389:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1924.358546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1928.903692] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1936.043201] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1941.122596] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1946.895291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1947.129698] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1947.404902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1953.611683] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1961.566824] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1963.727846] Lustre: 76965:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1965.541406] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1973.733080] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1973.735337] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1973.741339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1973.741346] Lustre: Skipped 3 previous similar messages [ 1973.750295] Lustre: Skipped 1 previous similar message [ 1973.776300] LustreError: Skipped 1 previous similar message [ 1977.036233] Lustre: server umount lustre-OST0000 complete [ 1983.972461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.977192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1983.998767] Lustre: Skipped 1 previous similar message [ 1984.014284] Lustre: Skipped 2 previous similar messages [ 1989.090182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1989.093453] Lustre: Skipped 1 previous similar message [ 1995.231449] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1995.506439] Lustre: server umount lustre-MDT0000 complete [ 1999.188859] LustreError: 76256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787600882 with bad export cookie 13029069235374585034 [ 1999.194214] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1999.198560] LustreError: 76256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1999.523616] Lustre: server umount lustre-MDT0001 complete [ 2012.515193] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 2015.595195] Key type lgssc unregistered [ 2015.890067] LNet: 78181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2015.896693] LNetError: 78181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2015.913503] LNet: Removed LNI 192.168.206.103@tcp [ 2016.783458] Key type .llcrypt unregistered [ 2016.785188] Key type ._llcrypt unregistered [ 2042.498507] Key type ._llcrypt registered [ 2042.500893] Key type .llcrypt registered [ 2042.584316] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 2043.461691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2043.563965] alg: No test for adler32 (adler32-zlib) [ 2044.694639] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 2044.912805] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 2046.592429] Key type lgssc registered [ 2047.513621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2054.795634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2054.833340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2056.430950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2060.515396] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2066.919656] LustreError: 79596:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2066.929502] LustreError: 79596:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2068.330475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2068.864596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2073.237118] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2081.148689] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2086.839956] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2093.808946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2094.333646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2100.676364] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2108.562672] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2114.385750] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2115.321454] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2117.084722] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2117.105102] Lustre: Skipped 1 previous similar message [ 2119.759940] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2119.767643] Lustre: Skipped 2 previous similar messages [ 2124.573477] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2124.583265] Lustre: Skipped 4 previous similar messages [ 2132.839078] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2132.845653] Lustre: Skipped 8 previous similar messages [ 2149.274430] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2149.278464] Lustre: Skipped 18 previous similar messages [ 2181.991927] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2181.997801] Lustre: Skipped 36 previous similar messages [ 2211.660337] Lustre: 87696:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2214.121550] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 15:51:36 (1787601096) [ 2215.363631] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2217.217317] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 15:51:39 (1787601099) [ 2223.586068] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2223.599722] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2223.609197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2224.609201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2224.610147] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2229.731103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2229.736697] Lustre: Skipped 2 previous similar messages [ 2234.853071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2234.863880] Lustre: Skipped 1 previous similar message [ 2236.383128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2236.509280] Lustre: server umount lustre-OST0000 complete [ 2243.043488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2243.055461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2243.078565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2245.101995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2248.663969] Lustre: server umount lustre-MDT0000 complete [ 2250.208889] LustreError: 79593:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2250.220152] LustreError: 79593:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2252.065874] LustreError: 82185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787601135 with bad export cookie 3746583178436841144 [ 2252.066560] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2252.081404] LustreError: 82185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2252.344154] Lustre: server umount lustre-MDT0001 complete [ 2263.321782] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 2265.754258] Key type lgssc unregistered [ 2266.092505] LNet: 88949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2266.104389] LNetError: 88949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2266.128711] LNet: Removed LNI 192.168.206.103@tcp [ 2267.162906] Key type .llcrypt unregistered [ 2267.165324] Key type ._llcrypt unregistered [ 2271.377388] Key type ._llcrypt registered [ 2271.379015] Key type .llcrypt registered [ 2271.464964] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2275.780508] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules [ 2276.890420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2276.978303] alg: No test for adler32 (adler32-zlib) [ 2278.031469] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 2278.245564] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 2279.927148] Key type lgssc registered [ 2281.316550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2282.576852] LustreError: 90152:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2301.349502] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 2304.763694] LustreError: 90152:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2314.724524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2314.762276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2316.147778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2322.259594] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2325.498733] LustreError: 91182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2325.519912] LustreError: 91182:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2330.594374] LustreError: 91181:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2330.784600] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 15:53:33 (1787601213) [ 2335.400789] LustreError: 91213:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2335.413117] LustreError: 91213:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2335.420903] LustreError: 91213:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2335.769318] Lustre: server umount lustre-MDT0000 complete [ 2349.409164] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 2351.971717] Key type lgssc unregistered [ 2352.236524] LNet: 92709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2352.244560] LNetError: 92709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2353.318283] LNet: Removed LNI 192.168.206.103@tcp [ 2354.143398] Key type .llcrypt unregistered [ 2354.147747] Key type ._llcrypt unregistered [ 2370.659607] Key type ._llcrypt registered [ 2370.662826] Key type .llcrypt registered [ 2370.790119] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 2371.769846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2371.840660] alg: No test for adler32 (adler32-zlib) [ 2372.844986] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 2373.084674] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 2374.783486] Key type lgssc registered [ 2375.919786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2383.497260] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2383.514698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2384.962097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2389.444766] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2395.105782] LustreError: 93935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2395.135054] LustreError: 93935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2397.654737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2398.245665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2403.012814] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2411.581403] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2417.237950] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2424.618061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2425.329228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2431.650547] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2438.842795] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2491.335656] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 15:56:13 (1787601373) [ 2492.809967] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2494.858403] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 15:56:16 (1787601376) [ 2500.576744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2500.594200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2500.609397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2504.163577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2504.166737] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2504.170862] Lustre: Skipped 1 previous similar message [ 2504.183279] Lustre: Skipped 1 previous similar message [ 2505.891932] Lustre: server umount lustre-MDT0000 complete [ 2509.284119] LustreError: 93930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2509.292186] LustreError: 93915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787601393 with bad export cookie 6667051951453822602 [ 2509.308039] LustreError: 93930:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2509.308687] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2509.350772] LustreError: 93915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2509.762422] Lustre: server umount lustre-MDT0001 complete [ 2524.211469] Lustre: server umount lustre-OST0000 complete [ 2535.285871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2535.645281] LustreError: 98229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2535.793218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2540.089941] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2541.036723] LustreError: 98230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2546.149166] LustreError: 98229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2548.146658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2548.770954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2553.064960] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2561.182494] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2567.991045] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2580.330431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2580.454742] LustreError: 100321:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2580.530841] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2580.539713] LustreError: 98219:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2580.564696] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2580.573062] LustreError: 100315:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2580.579340] LustreError: 100315:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2580.585344] LustreError: 100315:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2580.650981] Lustre: server umount lustre-OST0000 complete [ 2580.653213] LustreError: 100315:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2589.389841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2589.591808] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2589.599688] LustreError: 98219:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2589.616896] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2589.625742] LustreError: 100777:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2589.630956] LustreError: 100777:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2589.637046] LustreError: 100777:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2589.732537] Lustre: server umount lustre-OST0000 complete [ 2589.736897] LustreError: 100777:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2598.381761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2598.511819] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2598.515076] Lustre: Found index 0 for lustre-OST0000, updating log [ 2598.518970] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2598.593936] Lustre: lustre-OST0000: new disk, initializing [ 2598.597259] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2598.602558] Lustre: 101312:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2598.851042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2605.016372] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2608.793106] Lustre: Failing over lustre-OST0000 [ 2609.031475] Lustre: server umount lustre-OST0000 complete [ 2615.265391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2615.273806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2615.302886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2615.309489] Lustre: Skipped 1 previous similar message [ 2618.347131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2618.363975] Lustre: Skipped 3 previous similar messages [ 2619.221688] Lustre: server umount lustre-MDT0000 complete [ 2622.940201] LustreError: 101311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787601506 with bad export cookie 6667051951453825353 [ 2622.941553] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2622.955230] LustreError: 101311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2623.348892] Lustre: server umount lustre-MDT0001 complete [ 2631.056919] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 15:58:33 (1787601513) [ 2647.672556] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 2657.303044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2657.576630] LustreError: 103654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2657.664401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2661.418750] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2669.968129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2675.859913] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2678.670343] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2685.312745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2692.258868] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2700.786290] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2700.865522] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2700.953440] Lustre: lustre-OST0001: new disk, initializing [ 2700.961425] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2700.971956] Lustre: 105974:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2701.353286] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2701.359900] Lustre: Skipped 2 previous similar messages [ 2707.456304] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2710.115898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2710.131761] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2710.259199] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2715.316097] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2723.145358] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2730.598238] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.3@tcp (stopping) [ 2731.008996] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2731.044984] Lustre: Skipped 3 previous similar messages [ 2734.752951] Lustre: server umount lustre-MDT0001 complete [ 2735.695227] LustreError: 103649:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2735.731293] LustreError: 103649:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2747.140261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2747.224854] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2747.236615] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2747.288186] Lustre: 103642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2748.865492] LustreError: 107534:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2748.882373] LustreError: 107534:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2748.894926] LustreError: 103646:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2749.010423] LustreError: 107548:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.103@tcp: cfg command failed: rc = -17 [ 2749.017193] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2749.017193] [ 2752.664612] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2753.000699] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2762.505567] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 16:00:44 (1787601644) [ 2780.088524] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 2799.112886] Lustre: 109598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2819.530257] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2822.612511] Lustre: 110732:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2906.599109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2906.607569] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2906.608452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2906.608457] Lustre: Skipped 3 previous similar messages [ 2906.643951] Lustre: Skipped 3 previous similar messages [ 2911.375128] Lustre: server umount lustre-MDT0000 complete [ 2911.712599] LustreError: 103650:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2911.741530] LustreError: 103650:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 2915.737073] LustreError: 107465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787601799 with bad export cookie 6667051951453826277 [ 2915.739300] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2915.785468] LustreError: 107465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2916.258948] Lustre: server umount lustre-MDT0001 complete [ 2931.380600] Lustre: server umount lustre-OST0000 complete [ 2933.153174] Lustre: 93249:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787601800/real 1787601800] req@ffff9ff8b5561500 x1874435774964864/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787601816 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2933.191709] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2935.503935] Lustre: server umount lustre-OST0001 complete [ 2941.442470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2951.959917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2959.999923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2969.118436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2990.446876] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 2998.870984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2998.893165] Lustre: lustre-MDT0000: reset Object Index mappings [ 2999.247786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2999.255574] Lustre: Skipped 1 previous similar message [ 3003.204308] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3010.617837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3010.649561] Lustre: lustre-MDT0001: reset Object Index mappings [ 3016.035628] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3018.821116] Lustre: 115828:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3024.929349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3031.319297] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3035.888692] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3039.496856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3044.084819] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3044.146736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3044.187918] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3045.022387] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3052.073522] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3055.536387] Lustre: 117704:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3103.416873] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 16:06:25 (1787601985) [ 3133.920251] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3133.926498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3133.939083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3133.946108] Lustre: Skipped 3 previous similar messages [ 3137.963326] Lustre: server umount lustre-MDT0000 complete [ 3139.559158] LustreError: 114677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3139.587583] LustreError: 114677:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3141.608713] LustreError: 114661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787602025 with bad export cookie 6667051951453873149 [ 3141.622251] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3148.107996] Lustre: server umount lustre-MDT0001 complete [ 3158.238851] Lustre: server umount lustre-OST0000 complete [ 3167.709835] Lustre: server umount lustre-OST0001 complete [ 3173.589384] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 3181.974699] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 3213.370278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3213.658442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3213.699439] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3213.787875] Lustre: lustre-MDT0000: new disk, initializing [ 3213.852521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3213.856992] Lustre: Skipped 3 previous similar messages [ 3213.867269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3217.665347] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3223.385497] Lustre: server umount lustre-MDT0000 complete [ 3231.686114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3231.927365] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3231.947757] Lustre: 121972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3231.968637] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3237.410895] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3242.303256] Lustre: server umount lustre-MDT0000 complete [ 3250.526819] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 16:08:52 (1787602132) [ 3251.947471] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3253.876946] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 16:08:56 (1787602136) [ 3268.946822] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 3278.123973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3282.486468] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3291.652581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3291.894403] Lustre: 124020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3291.950815] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3291.955132] Lustre: Skipped 1 previous similar message [ 3292.047183] Lustre: lustre-MDT0001: new disk, initializing [ 3292.192264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3292.215917] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3297.403460] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3302.700475] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3310.648554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3310.991833] Lustre: lustre-OST0000: new disk, initializing [ 3310.997567] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3311.002698] Lustre: 125598:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3312.403179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3312.411631] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3312.509556] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3317.784653] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3328.741247] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3328.924926] Lustre: lustre-OST0001: new disk, initializing [ 3328.936875] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3328.947423] Lustre: 126468:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3330.092400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3330.109320] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3330.242721] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3336.489738] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3346.194456] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3349.496202] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3712.905502] Lustre: 132880:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3712.909748] Lustre: 132880:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3712.912720] Lustre: 132880:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3712.917560] Lustre: 132880:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3712.922488] Lustre: 132880:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3712.927594] Lustre: 132880:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3998.061671] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:21:19 (1787602879) [ 4014.564053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4014.571612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4014.581130] Lustre: Skipped 5 previous similar messages [ 4014.584617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4014.589647] Lustre: Skipped 8 previous similar messages [ 4018.476758] Lustre: server umount lustre-MDT0000 complete [ 4021.508339] LustreError: 124757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787602905 with bad export cookie 6667051951453940517 [ 4021.523889] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4021.526330] LustreError: 124757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4021.731193] LustreError: 127295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4021.732363] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4021.739901] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4021.739912] Lustre: Skipped 4 previous similar messages [ 4021.753269] LustreError: 127295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4021.785146] Lustre: Skipped 4 previous similar messages [ 4027.984848] Lustre: server umount lustre-MDT0001 complete [ 4037.297388] Lustre: server umount lustre-OST0000 complete [ 4047.575598] Lustre: server umount lustre-OST0001 complete [ 4053.932961] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 4062.403253] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 4103.601291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4103.843181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4103.880393] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4103.999721] Lustre: lustre-MDT0000: new disk, initializing [ 4104.120401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4104.124585] Lustre: Skipped 5 previous similar messages [ 4104.135740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4108.607898] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 4119.660494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4119.752747] Lustre: 135942:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4119.873519] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4119.884942] Lustre: Skipped 1 previous similar message [ 4119.977646] Lustre: lustre-MDT0001: new disk, initializing [ 4120.118936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4120.129072] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4124.776349] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 4133.603236] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4138.803209] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4145.039022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4145.348451] Lustre: lustre-OST0000: new disk, initializing [ 4145.351856] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4145.360558] Lustre: 137912:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4145.443901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4145.453362] Lustre: Skipped 1 previous similar message [ 4147.351821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4147.362436] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4147.538804] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4151.280431] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 4161.538608] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4172.455569] Lustre: lustre-MDD0000: changelog on [ 4175.675346] Lustre: lustre-MDD0001: changelog on [ 6019.707599] Lustre: lustre-MDD0001: changelog off [ 6021.646068] Lustre: 137704:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6022.517534] Lustre: lustre-MDD0000: changelog off [ 6029.798268] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6029.804228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6029.817781] Lustre: Skipped 1 previous similar message [ 6029.829519] Lustre: Skipped 4 previous similar messages [ 6034.918729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6034.932192] Lustre: Skipped 1 previous similar message [ 6040.037492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6040.057375] Lustre: Skipped 1 previous similar message [ 6044.127659] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6044.338542] Lustre: server umount lustre-OST0000 complete [ 6050.276414] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6050.280253] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6050.288824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6050.300357] Lustre: Skipped 1 previous similar message [ 6060.512616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6060.520318] Lustre: Skipped 2 previous similar messages [ 6062.047124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6062.263309] Lustre: server umount lustre-MDT0000 complete [ 6065.634698] LustreError: 136827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6065.674340] LustreError: 136827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6066.035523] LustreError: 136739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787604949 with bad export cookie 6667051951454759384 [ 6066.055936] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6066.390701] Lustre: server umount lustre-MDT0001 complete [ 6075.148575] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 6077.418820] Key type lgssc unregistered [ 6077.674694] LNet: 147051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6077.681163] LNetError: 147051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6077.695606] LNet: Removed LNI 192.168.206.103@tcp [ 6078.412639] Key type .llcrypt unregistered [ 6078.417313] Key type ._llcrypt unregistered [ 6097.042313] Key type ._llcrypt registered [ 6097.044798] Key type .llcrypt registered [ 6097.131321] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 6113.156165] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 6114.412912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6114.681725] alg: No test for adler32 (adler32-zlib) [ 6115.640356] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 6115.942327] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 6117.591173] Key type lgssc registered [ 6118.684216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6156.878859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6156.920303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6158.268070] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6158.324549] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6158.407059] Lustre: lustre-MDT0000: new disk, initializing [ 6158.490718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6158.519043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6164.314467] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6176.342685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6176.483976] Lustre: 150255:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6176.570964] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6176.585726] Lustre: Skipped 1 previous similar message [ 6176.682693] Lustre: lustre-MDT0001: new disk, initializing [ 6176.807123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6176.863452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6176.900779] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6181.877601] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6192.079805] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6198.567521] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6208.072783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6208.768494] Lustre: lustre-OST0000: new disk, initializing [ 6208.793932] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6208.803509] Lustre: 152225:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6208.996310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6216.882852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6216.907611] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6217.086521] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6220.351267] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6232.728090] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6239.580351] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_module obdecho/obdecho [ 6242.291965] LustreError: 153231:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6242.304395] LustreError: 153231:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6244.826169] LustreError: 153327:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6263.265696] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6263.279029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6263.282411] Lustre: Skipped 2 previous similar messages [ 6263.285825] Lustre: Skipped 1 previous similar message [ 6265.324469] Lustre: server umount lustre-MDT0000 complete [ 6269.245931] LustreError: 151059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787605153 with bad export cookie 9654565905066598344 [ 6269.254154] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6269.259464] LustreError: 151059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6269.671033] Lustre: server umount lustre-MDT0001 complete [ 6284.306278] Lustre: server umount lustre-OST0000 complete [ 6307.767456] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 6316.466739] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 6350.002962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6350.210738] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6350.228389] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6350.309874] Lustre: lustre-MDT0000: new disk, initializing [ 6350.391182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6350.407500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6354.983412] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6365.506931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6365.657843] Lustre: 157040:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6365.705570] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6365.709983] Lustre: Skipped 1 previous similar message [ 6365.772087] Lustre: lustre-MDT0001: new disk, initializing [ 6365.872839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6365.938346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6365.951053] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6370.733153] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6381.627527] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6388.585223] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6395.885952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6396.304541] Lustre: lustre-OST0000: new disk, initializing [ 6396.307420] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6396.311617] Lustre: 159022:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6396.401660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6403.996881] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6405.757274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6405.779793] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6405.935317] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6414.813157] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6419.710758] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6419.941487] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6424.522860] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6424.748697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6426.593988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6426.596566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6426.601956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6426.601964] Lustre: Skipped 1 previous similar message [ 6426.615023] Lustre: Skipped 1 previous similar message [ 6426.636038] LustreError: Skipped 1 previous similar message [ 6431.716480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6431.721691] Lustre: Skipped 2 previous similar messages [ 6432.584767] Lustre: server umount lustre-OST0000 complete [ 6436.832937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6436.835626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6436.847801] Lustre: Skipped 1 previous similar message [ 6436.868488] Lustre: Skipped 1 previous similar message [ 6441.961030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6441.971511] Lustre: Skipped 1 previous similar message [ 6442.201070] Lustre: server umount lustre-MDT0000 complete [ 6445.530784] LustreError: 157033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787605329 with bad export cookie 9654565905066599296 [ 6445.534480] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6445.556458] LustreError: 157033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6445.813590] Lustre: server umount lustre-MDT0001 complete [ 6449.194923] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 17:02:11 (1787605331) [ 6450.720912] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6452.390235] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 17:02:14 (1787605334) [ 6459.340849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6459.727182] LustreError: 161435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6459.747473] LustreError: 161435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6459.831461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6463.746767] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6464.992787] LustreError: 161436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6470.119689] LustreError: 161435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6471.761891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6477.634623] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6484.371960] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6489.463361] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6494.617649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6494.970154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6494.975677] Lustre: Skipped 1 previous similar message [ 6500.740840] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6507.300361] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6513.125562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6513.149102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6513.160286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6514.608385] Lustre: server umount lustre-MDT0000 complete [ 6515.681277] LustreError: 162163:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6515.712470] LustreError: 162163:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6518.260589] LustreError: 161417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787605402 with bad export cookie 9654565905066600178 [ 6518.272162] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6518.280658] LustreError: 161417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6520.804604] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6520.806950] LustreError: 161430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6520.815614] Lustre: Skipped 2 previous similar messages [ 6520.823298] LustreError: 161430:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6524.708101] Lustre: server umount lustre-MDT0001 complete [ 6531.142445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6543.493532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6543.697907] LustreError: 165273:0:(llog.c:1655:llog_backup()) MGC192.168.206.103@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6543.702514] Lustre: 165273:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.103@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6548.449706] LustreError: 165278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6548.464314] LustreError: 165278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6568.928167] LustreError: 165279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6568.952486] LustreError: 165279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 6584.382988] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.3@tcp (not set up) [ 6584.389679] Lustre: Skipped 5 previous similar messages [ 6603.841527] LustreError: 165278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6603.863192] LustreError: 165278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 23 previous similar messages [ 6604.767376] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6604.778893] Lustre: 165273:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.103@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6645.779703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6649.727229] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6650.865236] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6661.156530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6663.521197] LustreError: 166021:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 6663.529090] LustreError: 166021:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@000000005f063c35 x1874439699556992/t0(0) o250->MGC192.168.206.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1787605547 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6663.569729] LustreError: 166021:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 6664.255236] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.3@tcp (not set up) [ 6669.808288] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6671.954897] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6680.627766] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6682.021554] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6686.615989] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6688.520510] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6705.637831] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6705.640954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6712.271394] Lustre: server umount lustre-MDT0000 complete [ 6715.876661] LustreError: 165278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6715.904509] LustreError: 165278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 46 previous similar messages [ 6717.530268] LustreError: 161415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787605601 with bad export cookie 9654565905066606639 [ 6717.533946] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6717.553250] LustreError: 161415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6717.969324] Lustre: server umount lustre-MDT0001 complete [ 6732.925752] Lustre: server umount lustre-OST0000 complete [ 6753.021096] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 6761.882971] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 6795.278783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6795.464856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6795.487320] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6795.541673] Lustre: lustre-MDT0000: new disk, initializing [ 6795.590704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6795.599647] Lustre: Skipped 1 previous similar message [ 6795.609695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6799.483086] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6808.885859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6809.013372] Lustre: 170383:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6809.248755] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6813.243213] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6822.062355] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6828.866971] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6837.269387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6837.598811] Lustre: lustre-OST0000: new disk, initializing [ 6837.602434] Lustre: Skipped 1 previous similar message [ 6837.607217] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6837.609404] Lustre: Skipped 2 previous similar messages [ 6837.622524] Lustre: 172353:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6844.517927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6844.525434] Lustre: Skipped 1 previous similar message [ 6844.531615] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6844.693042] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6844.962965] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6855.637826] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6860.422527] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6860.607875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6865.167919] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6865.470558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6870.498532] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6870.500050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6870.514208] LustreError: Skipped 1 previous similar message [ 6870.514932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6870.514937] Lustre: Skipped 6 previous similar messages [ 6870.552983] Lustre: Skipped 3 previous similar messages [ 6873.665459] Lustre: server umount lustre-OST0000 complete [ 6884.786720] Lustre: server umount lustre-MDT0000 complete [ 6885.873495] LustreError: 170390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6885.901281] LustreError: 170390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6888.308684] LustreError: 170375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787605772 with bad export cookie 9654565905066623474 [ 6888.316109] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6888.323574] LustreError: 170375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6888.820951] Lustre: server umount lustre-MDT0001 complete [ 6897.781193] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 17:09:39 (1787605779) [ 6903.863335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6908.782285] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6916.545376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6921.677974] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6928.569738] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6934.020781] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6940.354404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6947.056879] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6955.041157] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6963.895797] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 17:10:45 (1787605845) [ 6970.336407] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6970.340329] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6970.372922] Lustre: Skipped 3 previous similar messages [ 6973.984167] Lustre: server umount lustre-OST0000 complete [ 6978.533049] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6983.774567] Lustre: server umount lustre-MDT0000 complete [ 6987.011169] LustreError: 174746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787605870 with bad export cookie 9654565905066624356 [ 6987.019958] LustreError: 174746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6987.022144] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6987.306954] Lustre: server umount lustre-MDT0001 complete [ 6996.710777] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 6999.660861] Key type lgssc unregistered [ 7000.077545] LNet: 178492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7000.083457] LNetError: 178492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7000.098073] LNet: Removed LNI 192.168.206.103@tcp [ 7001.048889] Key type .llcrypt unregistered [ 7001.051717] Key type ._llcrypt unregistered [ 7023.134357] Key type ._llcrypt registered [ 7023.141911] Key type .llcrypt registered [ 7023.270908] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 7024.777192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7024.897227] alg: No test for adler32 (adler32-zlib) [ 7026.466756] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 7027.076153] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 7028.879177] Key type lgssc registered [ 7030.130769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7040.333085] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7040.359310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7056.095635] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7061.216994] LustreError: 179739:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.103@tcp: failed processing log, type 4: rc = -110 [ 7066.335701] LustreError: 179739:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7066.359185] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7066.367669] LustreError: MGC192.168.206.103@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7066.374376] LustreError: 179739:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7066.382318] LustreError: 179739:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 7066.389574] LustreError: 179739:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7066.456559] Lustre: server umount lustre-OST0000 complete [ 7066.459569] LustreError: 179739:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7074.210553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7075.683880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7079.981402] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7086.058092] LustreError: 180238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7086.089019] LustreError: 180238:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7088.820111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7089.215093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7093.589501] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7102.164782] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7107.532907] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7114.993553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7115.140756] LustreError: 182056:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7115.162735] Lustre: 182056:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.103@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7115.321892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7122.189466] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7129.878492] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7134.117756] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7134.124835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7134.139403] Lustre: Skipped 2 previous similar messages [ 7134.178957] Lustre: Skipped 2 previous similar messages [ 7138.527928] Lustre: server umount lustre-MDT0000 complete [ 7139.297474] LustreError: 180238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7139.344294] LustreError: 180238:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7142.972402] LustreError: 180218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787606026 with bad export cookie 11912812780022578845 [ 7142.987268] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7143.010621] LustreError: 180218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7144.430239] LustreError: 180233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7144.430511] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7144.471484] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7144.480386] LustreError: 180968:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000162804f1 x1874440654713728/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7144.708453] Lustre: server umount lustre-MDT0001 complete [ 7150.013376] Lustre: server umount lustre-OST0000 complete [ 7160.546740] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 7170.424862] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 7206.918498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7207.210038] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7207.254362] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7207.363066] Lustre: lustre-MDT0000: new disk, initializing [ 7207.446926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7207.463411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7212.104371] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7223.283901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7223.453316] Lustre: 185580:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7223.527445] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7223.532213] Lustre: Skipped 1 previous similar message [ 7223.665840] Lustre: lustre-MDT0001: new disk, initializing [ 7223.787475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7223.827597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7223.839387] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7228.172928] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7237.088821] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7244.070954] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7251.931702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7252.124551] Lustre: lustre-OST0000: new disk, initializing [ 7252.130690] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7252.140206] Lustre: 187549:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7252.243618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7258.131554] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7260.233343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7260.254347] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7260.372680] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7267.292960] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7271.681897] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7271.893993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7277.292952] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7277.585945] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7281.128510] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7281.131086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7281.156585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7281.178550] Lustre: Skipped 1 previous similar message [ 7286.101047] Lustre: server umount lustre-OST0000 complete [ 7290.336439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7290.343298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7290.355276] Lustre: Skipped 1 previous similar message [ 7290.364820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7296.432796] Lustre: server umount lustre-MDT0000 complete [ 7296.480691] LustreError: 186711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7296.513738] LustreError: 186711:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7300.181861] LustreError: 186375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787606183 with bad export cookie 11912812780022579566 [ 7300.187047] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7300.190934] LustreError: 186375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7300.495502] Lustre: server umount lustre-MDT0001 complete [ 7309.191691] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 17:16:31 (1787606191) [ 7330.884237] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 7334.673951] Key type lgssc unregistered [ 7335.012618] LNet: 190496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7335.019356] LNetError: 190496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7335.037273] LNet: Removed LNI 192.168.206.103@tcp [ 7336.082433] Key type .llcrypt unregistered [ 7336.086355] Key type ._llcrypt unregistered [ 7356.559468] Key type ._llcrypt registered [ 7356.562856] Key type .llcrypt registered [ 7356.744593] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 7358.430769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7358.513091] alg: No test for adler32 (adler32-zlib) [ 7360.022164] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 7360.414664] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 7362.287226] Key type lgssc registered [ 7364.319995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7373.639838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7373.663747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7375.303354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7380.496146] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7385.581317] LustreError: 191716:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7385.613811] LustreError: 191716:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7390.532826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7390.690653] LustreError: 191717:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7391.116335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7395.872640] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7404.898612] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7411.212934] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7419.173092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7419.492486] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7419.501246] LustreError: 193580:0:(llog.c:1696:llog_backup()) MGC192.168.206.103@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7419.518980] Lustre: 193580:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.103@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7419.669163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7425.983216] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7433.787590] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7436.270201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7436.275976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7436.284866] Lustre: Skipped 1 previous similar message [ 7436.299938] Lustre: Skipped 2 previous similar messages [ 7437.352834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7437.360319] Lustre: Skipped 2 previous similar messages [ 7442.275616] Lustre: server umount lustre-MDT0000 complete [ 7442.408862] LustreError: 191710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7442.428196] LustreError: 191710:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7446.922200] LustreError: 191695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787606330 with bad export cookie 13147552535232096360 [ 7446.932503] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7446.944814] LustreError: 191695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7447.520931] LustreError: 191711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7447.527163] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7447.547688] Lustre: Skipped 1 previous similar message [ 7447.554990] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7447.607634] LustreError: 192446:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000fa1144f8 x1874441004937856/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7447.792498] Lustre: server umount lustre-MDT0001 complete [ 7454.138827] Lustre: server umount lustre-OST0000 complete [ 7464.986771] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 7476.113143] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 7513.336137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7513.578178] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7513.634508] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7513.749683] Lustre: lustre-MDT0000: new disk, initializing [ 7513.818070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7513.845896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7518.036510] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7528.742834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7528.854317] Lustre: 197105:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7528.895733] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7528.898798] Lustre: Skipped 1 previous similar message [ 7528.901681] LustreError: 197889:0:(llog.c:1696:llog_backup()) MGC192.168.206.103@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7528.907355] LustreError: 197889:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7528.911516] Lustre: 197889:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.103@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7528.925625] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7528.929086] Lustre: Skipped 1 previous similar message [ 7529.034635] Lustre: lustre-MDT0001: new disk, initializing [ 7529.141570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7529.222690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7529.228697] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7534.399858] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7545.458553] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7552.476903] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7559.971573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7560.274093] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7560.283144] Lustre: Skipped 1 previous similar message [ 7560.287438] LustreError: 199051:0:(llog.c:1696:llog_backup()) MGC192.168.206.103@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7560.305762] LustreError: 199051:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7560.314787] Lustre: 199051:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.103@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7560.326270] Lustre: 199051:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7560.380090] Lustre: lustre-OST0000: new disk, initializing [ 7560.395145] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7560.411045] Lustre: 199075:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7560.569628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7567.165812] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7567.953763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7567.970817] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7568.172861] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7577.453568] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7582.817349] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7583.075800] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7588.227083] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7588.421786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7593.959241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7593.963805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7593.968155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7593.986537] LustreError: Skipped 1 previous similar message [ 7593.990489] Lustre: Skipped 1 previous similar message [ 7596.564434] Lustre: server umount lustre-OST0000 complete [ 7601.130155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7601.141863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7601.158535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7601.164562] Lustre: Skipped 1 previous similar message [ 7606.553368] Lustre: server umount lustre-MDT0000 complete [ 7609.320398] LustreError: 197114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7609.344299] LustreError: 197114:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7609.992908] LustreError: 197098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787606493 with bad export cookie 13147552535232097081 [ 7610.009787] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7610.013372] LustreError: 197098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7610.485475] Lustre: server umount lustre-MDT0001 complete [ 7619.080714] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 17:21:41 (1787606501) [ 7634.221572] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 7643.294828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7643.671867] LustreError: 202204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7643.726703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7647.673302] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7655.793215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7660.529359] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7663.462698] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7670.804502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7677.918621] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7686.823989] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7687.024856] Lustre: lustre-OST0001: new disk, initializing [ 7687.031088] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7687.039752] Lustre: 204511:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7687.176730] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7687.178824] Lustre: Skipped 2 previous similar messages [ 7692.400527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7692.426567] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7692.535054] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7694.300286] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7707.074066] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7713.138635] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7715.109071] Lustre: DEBUG MARKER: ADD OST3 [ 7720.244674] loop: module loaded [ 7720.356475] loop0: detected capacity change from 0 to 204800000 [ 7720.505480] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7720.525255] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7720.545033] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7726.631942] loop0: detected capacity change from 0 to 204800000 [ 7727.506188] Lustre: DEBUG MARKER: STOP OST3 [ 7731.976126] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7732.265867] Lustre: lustre-OST0003: new disk, initializing [ 7732.279205] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7732.294480] Lustre: 206503:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7733.878487] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7734.038484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7734.045765] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7734.095872] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7735.911318] LustreError: 206650:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7736.307613] Lustre: Failing over lustre-OST0003 [ 7736.403513] Lustre: server umount lustre-OST0003 complete [ 7738.462136] LustreError: 203691:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7738.482698] LustreError: 203691:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7738.631623] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7739.373685] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7739.385483] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7739.404590] Lustre: Skipped 1 previous similar message [ 7747.281808] Lustre: DEBUG MARKER: START OST3 again [ 7753.421245] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7753.703055] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7753.708583] Lustre: Skipped 1 previous similar message [ 7753.722598] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7753.795932] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7754.895962] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7754.897204] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7759.924297] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7779.069466] Lustre: Failing over lustre-OST0003 [ 7779.219680] Lustre: server umount lustre-OST0003 complete [ 7779.314315] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7779.322754] LustreError: 204797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7779.330045] Lustre: Skipped 2 previous similar messages [ 7779.361871] LustreError: 204797:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 7799.777409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7799.793146] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7799.797499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7799.797507] Lustre: Skipped 2 previous similar messages [ 7799.834948] Lustre: Skipped 3 previous similar messages [ 7802.231496] Lustre: server umount lustre-MDT0000 complete [ 7806.843909] LustreError: 202186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787606690 with bad export cookie 13147552535232097963 [ 7806.888907] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7816.161662] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7816.173878] Lustre: Skipped 8 previous similar messages [ 7821.279216] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7821.482436] Lustre: server umount lustre-MDT0001 complete [ 7825.991687] Lustre: server umount lustre-OST0000 complete [ 7830.118931] Lustre: server umount lustre-OST0001 complete [ 7835.946286] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 7843.482717] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 7877.377946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7877.725169] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7877.781817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7877.924214] Lustre: lustre-MDT0000: new disk, initializing [ 7878.047776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7882.603994] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7893.239775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7893.364827] Lustre: 211978:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7893.558967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7893.565648] Lustre: Skipped 1 previous similar message [ 7893.595838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7898.201478] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7907.078290] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7911.813118] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7917.624944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7917.937558] Lustre: lustre-OST0000: new disk, initializing [ 7917.942483] Lustre: Skipped 1 previous similar message [ 7917.945847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7917.949443] Lustre: Skipped 2 previous similar messages [ 7917.954274] Lustre: 213945:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7919.239326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7919.251047] Lustre: Skipped 1 previous similar message [ 7919.253942] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7919.347291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7923.517968] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7931.549155] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7935.615181] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7935.827280] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7939.773603] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7939.982155] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7945.184784] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7945.186096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7945.191743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7945.191750] Lustre: Skipped 2 previous similar messages [ 7945.204941] Lustre: Skipped 3 previous similar messages [ 7948.110732] Lustre: server umount lustre-OST0000 complete [ 7954.912138] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7966.175127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7966.367156] Lustre: server umount lustre-MDT0000 complete [ 7970.065568] LustreError: 211969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787606853 with bad export cookie 13147552535232100259 [ 7970.067804] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7970.083279] LustreError: 211969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7970.541328] Lustre: server umount lustre-MDT0001 complete [ 7974.709403] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 17:27:36 (1787606856) [ 7992.537241] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 8002.003286] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 8037.023041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8037.264585] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8037.307444] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8037.425522] Lustre: lustre-MDT0000: new disk, initializing [ 8037.518882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8042.029939] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8052.226699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8052.346453] Lustre: 218556:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8052.617751] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8057.132258] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8066.487481] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8071.701449] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8078.961901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8079.300522] Lustre: 220521:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8081.412628] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8086.156305] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8096.590585] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8101.772198] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8101.983742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8106.683857] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8106.910449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8112.097547] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8112.098115] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8112.105836] LustreError: Skipped 1 previous similar message [ 8112.108456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8112.108464] Lustre: Skipped 8 previous similar messages [ 8112.146457] Lustre: Skipped 3 previous similar messages [ 8114.756669] Lustre: server umount lustre-OST0000 complete [ 8124.186798] Lustre: server umount lustre-MDT0000 complete [ 8127.461768] LustreError: 218567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8127.495961] LustreError: 218567:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 21 previous similar messages [ 8128.092977] LustreError: 218548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787607011 with bad export cookie 13147552535232101141 [ 8128.097166] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8128.108631] LustreError: 218548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8128.455480] Lustre: server umount lustre-MDT0001 complete [ 8135.729826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8140.490371] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8149.824531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8150.237993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8150.250205] Lustre: Skipped 5 previous similar messages [ 8154.957373] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8162.380314] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8168.066393] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8174.092056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8180.317276] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8186.704332] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8196.069062] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8196.079865] LustreError: Skipped 1 previous similar message [ 8209.887714] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8210.067508] Lustre: server umount lustre-OST0000 complete [ 8219.931691] Lustre: server umount lustre-MDT0000 complete [ 8223.464609] LustreError: 222723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787607107 with bad export cookie 13147552535232102023 [ 8223.480181] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8223.480992] LustreError: 222723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8223.766729] Lustre: server umount lustre-MDT0001 complete [ 8326.971715] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 8329.796614] Key type lgssc unregistered [ 8330.039722] LNet: 226406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8330.052207] LNetError: 226406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8330.073211] LNet: Removed LNI 192.168.206.103@tcp [ 8331.112202] Key type .llcrypt unregistered [ 8331.115819] Key type ._llcrypt unregistered [ 8359.591647] Key type ._llcrypt registered [ 8359.594150] Key type .llcrypt registered [ 8359.681248] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 8375.595333] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 8376.731700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8376.749540] alg: No test for adler32 (adler32-zlib) [ 8377.781918] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 8378.032526] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 8379.687158] Key type lgssc registered [ 8380.622368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8416.179737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8416.205626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8417.620615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8417.696524] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8417.778889] Lustre: lustre-MDT0000: new disk, initializing [ 8417.857988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8417.877638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8423.771675] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8434.030648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8434.129970] Lustre: 229759:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8434.251501] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8434.261716] Lustre: Skipped 1 previous similar message [ 8434.382528] Lustre: lustre-MDT0001: new disk, initializing [ 8434.549803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8434.614395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8434.628656] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8439.000338] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8447.861269] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8453.964589] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8460.597822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8460.970328] Lustre: lustre-OST0000: new disk, initializing [ 8460.979372] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8460.999894] Lustre: 231729:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8461.095203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8467.222365] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8470.591268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8470.613797] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8470.740100] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8477.295297] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8482.273557] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8482.561252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8488.427245] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8488.625410] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8491.510096] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8491.521095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8491.534436] LustreError: Skipped 1 previous similar message [ 8491.535064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8491.551729] Lustre: Skipped 1 previous similar message [ 8496.623134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8496.631672] Lustre: Skipped 1 previous similar message [ 8501.736660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8501.748206] Lustre: Skipped 2 previous similar messages [ 8505.311273] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8505.517429] Lustre: server umount lustre-OST0000 complete [ 8511.457191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8511.467265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8511.482229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8515.496302] Lustre: server umount lustre-MDT0000 complete [ 8519.094222] LustreError: 229751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787607402 with bad export cookie 1965663804376526952 [ 8519.097625] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8519.119447] LustreError: 229751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8519.439415] Lustre: server umount lustre-MDT0001 complete [ 8536.546779] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 8546.038841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8546.325571] LustreError: 234664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8546.342459] LustreError: 234664:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8546.372207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8550.770562] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8551.392652] LustreError: 234665:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8556.516500] LustreError: 234664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8558.886664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8559.267989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8563.279324] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8565.734563] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8571.885736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8572.261781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8577.562253] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8585.566798] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8585.734979] Lustre: lustre-OST0001: new disk, initializing [ 8585.738936] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8585.745154] Lustre: 236973:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8591.405148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8591.417482] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8591.507905] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8591.623330] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8602.450296] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8612.475476] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8625.880415] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 17:38:27 (1787607507) [ 8628.717110] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8628.727597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8628.738869] Lustre: Skipped 4 previous similar messages [ 8628.748819] Lustre: Skipped 3 previous similar messages [ 8635.152531] Lustre: server umount lustre-MDT0000 complete [ 8638.957989] LustreError: 236015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8638.999947] LustreError: 236015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8639.092713] LustreError: 234644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787607522 with bad export cookie 1965663804376527834 [ 8639.113427] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8639.116468] LustreError: 234644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8639.427862] Lustre: server umount lustre-MDT0001 complete [ 8654.747397] Lustre: server umount lustre-OST0000 complete [ 8667.598967] Lustre: server umount lustre-OST0001 complete [ 8673.992662] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 8682.737260] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 8717.012583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8717.252082] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8717.286967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8717.373860] Lustre: lustre-MDT0000: new disk, initializing [ 8717.441738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8717.447611] Lustre: Skipped 1 previous similar message [ 8717.477550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8722.037526] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8733.321739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8733.496215] Lustre: 240914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8733.637988] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8733.642699] Lustre: Skipped 1 previous similar message [ 8733.770492] Lustre: lustre-MDT0001: new disk, initializing [ 8734.040638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8734.063893] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8738.572809] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8747.371665] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8753.907420] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8761.707977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8762.008695] Lustre: lustre-OST0000: new disk, initializing [ 8762.014320] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8762.029720] Lustre: 242883:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8762.127037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8762.133316] Lustre: Skipped 1 previous similar message [ 8763.120214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8763.128850] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8763.275177] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8768.855400] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8777.983839] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8783.835281] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8784.141325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8789.272935] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8789.517882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8794.080445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8794.091216] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8794.091290] LustreError: Skipped 1 previous similar message [ 8794.098806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8794.100861] Lustre: Skipped 1 previous similar message [ 8794.115625] Lustre: Skipped 5 previous similar messages [ 8797.940924] Lustre: server umount lustre-OST0000 complete [ 8804.321648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8804.335934] Lustre: Skipped 1 previous similar message [ 8808.358509] Lustre: server umount lustre-MDT0000 complete [ 8809.452789] LustreError: 241716:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8809.493320] LustreError: 241716:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8813.242340] LustreError: 240907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787607697 with bad export cookie 1965663804376529969 [ 8813.245266] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8813.259953] LustreError: 240907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8813.764736] Lustre: server umount lustre-MDT0001 complete [ 8822.767184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8823.216830] LustreError: 245099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8827.594992] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8836.362230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8836.848445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8836.861520] Lustre: Skipped 1 previous similar message [ 8841.331404] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8848.507314] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8853.776970] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8860.103829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8865.845896] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8872.669907] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9020.389152] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9020.407724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9020.430572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9020.436746] Lustre: Skipped 2 previous similar messages [ 9024.480396] Lustre: server umount lustre-OST0000 complete [ 9031.650203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9031.655845] LustreError: Skipped 1 previous similar message [ 9042.399554] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9042.563794] Lustre: server umount lustre-MDT0000 complete [ 9042.914492] LustreError: 245828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9042.932932] LustreError: 245828:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9046.351744] LustreError: 245080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787607930 with bad export cookie 1965663804376530844 [ 9046.369199] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9046.379264] LustreError: 245080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9046.700838] Lustre: server umount lustre-MDT0001 complete [ 9141.820849] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 9144.454378] Key type lgssc unregistered [ 9144.735847] LNet: 248711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9144.748012] LNetError: 248711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9144.776098] LNet: Removed LNI 192.168.206.103@tcp [ 9145.641328] Key type .llcrypt unregistered [ 9145.648229] Key type ._llcrypt unregistered [ 9171.723248] Key type ._llcrypt registered [ 9171.725072] Key type .llcrypt registered [ 9171.830980] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 9186.961480] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 9187.974984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9188.027308] alg: No test for adler32 (adler32-zlib) [ 9189.227056] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 9189.476369] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 9191.167689] Key type lgssc registered [ 9192.066856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9236.342565] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 9246.194936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9246.219786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9247.471887] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9247.557673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9247.670476] Lustre: lustre-MDT0000: new disk, initializing [ 9247.807820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9247.831744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9252.421775] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9262.865758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9263.036609] Lustre: 252783:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9263.170742] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9263.179898] Lustre: Skipped 1 previous similar message [ 9263.285854] Lustre: lustre-MDT0001: new disk, initializing [ 9263.385333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9263.449477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9263.464975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9268.096764] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9273.502827] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9281.080102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9281.422798] Lustre: lustre-OST0000: new disk, initializing [ 9281.427232] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9281.438032] Lustre: 254415:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9281.529450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9287.117079] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9287.757944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9287.767402] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9287.870325] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9297.157456] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9297.417499] Lustre: lustre-OST0001: new disk, initializing [ 9297.421870] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9297.430935] Lustre: 255284:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9297.511827] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9303.451665] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9306.172639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9306.178788] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9306.257329] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9313.209881] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9320.260406] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9335.264586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9335.271680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9335.286358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9336.804073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9336.806673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9336.835722] Lustre: Skipped 2 previous similar messages [ 9336.852116] Lustre: Skipped 3 previous similar messages [ 9338.578707] Lustre: server umount lustre-MDT0000 complete [ 9342.463598] LustreError: 256109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787608226 with bad export cookie 7722625213046566645 [ 9342.480903] LustreError: 256109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9342.489237] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9342.854144] Lustre: server umount lustre-MDT0001 complete [ 9356.356709] Lustre: server umount lustre-OST0000 complete [ 9370.338819] Lustre: server umount lustre-OST0001 complete [ 9383.203895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9392.769352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9401.838875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9411.527479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9431.072317] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 9443.503123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9443.521303] Lustre: lustre-MDT0000: reset Object Index mappings [ 9443.914222] LustreError: 260297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9443.931533] LustreError: 260297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9444.017157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9449.441419] LustreError: 260298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9449.498105] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9453.537613] LustreError: 260297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9458.675632] LustreError: 260298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9458.716842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9458.757628] Lustre: lustre-MDT0001: reset Object Index mappings [ 9459.149580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9463.339390] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9466.370488] Lustre: 261438:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9473.135039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9480.908503] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9484.014836] LustreError: 261793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9489.151741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9492.849983] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9493.109652] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9493.121051] Lustre: Skipped 1 previous similar message [ 9498.627296] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9500.504409] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9508.343235] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9511.689830] Lustre: 263312:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9518.494519] Lustre: server umount lustre-MDT0000 complete [ 9519.078103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9519.089691] LustreError: 261803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9519.102687] Lustre: Skipped 2 previous similar messages [ 9519.122222] LustreError: 261803:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9521.977818] LustreError: 260327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787608405 with bad export cookie 7722625213046573435 [ 9521.985107] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9521.992016] LustreError: 260327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9522.643055] Lustre: server umount lustre-MDT0001 complete [ 9536.820941] Lustre: server umount lustre-OST0000 complete [ 9540.064326] Lustre: 250290:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787608407/real 1787608407] req@ffff9ff78cb1c700 x1874442922880256/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787608423 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9540.102492] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9540.114611] Lustre: Skipped 1 previous similar message [ 9540.748970] Lustre: server umount lustre-OST0001 complete [ 9559.036492] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 17:54:01 (1787608441) [ 9573.889730] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 9583.670865] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 9618.453841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9618.686249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9618.717329] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9618.818862] Lustre: lustre-MDT0000: new disk, initializing [ 9618.943031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9618.955400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9623.742732] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9634.863935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9635.033731] Lustre: 267561:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9635.102664] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9635.112229] Lustre: Skipped 1 previous similar message [ 9635.246326] Lustre: lustre-MDT0001: new disk, initializing [ 9635.500123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9635.519326] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9640.629277] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9648.982173] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9653.742502] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9660.143200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9660.453942] Lustre: lustre-OST0000: new disk, initializing [ 9660.457876] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9660.464496] Lustre: 269534:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9661.556720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9661.576358] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9661.706773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9665.891656] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9674.443392] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9679.088648] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9679.308911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9683.960254] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9684.220228] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9687.523126] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9687.526138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9687.550258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9687.571266] Lustre: Skipped 1 previous similar message [ 9692.641896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9697.765620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9697.780762] Lustre: Skipped 2 previous similar messages [ 9700.319369] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9700.470355] Lustre: server umount lustre-OST0000 complete [ 9706.977104] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9706.991411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9707.012831] Lustre: Skipped 1 previous similar message [ 9707.024996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9710.315615] Lustre: server umount lustre-MDT0000 complete [ 9713.123432] LustreError: 268421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9713.141781] LustreError: 268421:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9714.680985] LustreError: 267556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787608598 with bad export cookie 7722625213046575528 [ 9714.691040] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9714.696145] LustreError: 267556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9715.219858] Lustre: server umount lustre-MDT0001 complete [ 9734.439724] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 9744.992410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9745.620347] LustreError: 272469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9745.747756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9745.752918] Lustre: Skipped 2 previous similar messages [ 9750.328878] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9760.919344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9761.257604] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9761.273434] Lustre: Skipped 2 previous similar messages [ 9766.608606] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9769.485582] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9775.636289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9782.223674] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9790.237451] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9790.393131] Lustre: lustre-OST0001: new disk, initializing [ 9790.398509] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9790.407145] Lustre: 274779:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9791.967706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9791.976413] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9792.108042] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9796.809071] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9805.311872] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9808.104573] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9812.679148] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9812.916783] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9822.702625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9822.713181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9822.727911] Lustre: Skipped 4 previous similar messages [ 9822.738902] Lustre: Skipped 3 previous similar messages [ 9825.610575] Lustre: server umount lustre-MDT0000 complete [ 9827.813164] LustreError: 275505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9827.827499] LustreError: 275505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9829.008392] LustreError: 273603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787608712 with bad export cookie 7722625213046576410 [ 9829.014551] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9829.029299] LustreError: 273603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9829.304626] Lustre: server umount lustre-MDT0001 complete [ 9842.324628] Lustre: server umount lustre-OST0000 complete [ 9856.422380] Lustre: server umount lustre-OST0001 complete [ 9874.755038] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 9884.984026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9885.458404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9885.467853] Lustre: Skipped 3 previous similar messages [ 9889.604597] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9898.068934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9902.751209] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9906.018510] Lustre: 278979:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9913.490537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9920.712674] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9929.681886] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9930.953249] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9930.965769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9935.930490] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9944.064720] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9948.581443] Lustre: 280857:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9955.162874] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9955.177943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9967.696603] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 18:00:49 (1787608849) [ 9983.312092] Lustre: server umount lustre-MDT0000 complete [ 9985.517088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9985.535739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9985.566828] LustreError: 277839:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9985.593678] LustreError: 277839:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9987.475780] LustreError: 277819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787608871 with bad export cookie 7722625213046582878 [ 9987.488205] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9987.496571] LustreError: 277819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9987.566025] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9990.259844] Lustre: server umount lustre-MDT0001 complete [10001.427337] Lustre: server umount lustre-OST0000 complete [10005.107319] Lustre: server umount lustre-OST0001 complete [10011.441830] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [10021.550081] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [10066.327519] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [10076.331556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10076.684139] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10076.723994] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10076.816687] Lustre: lustre-MDT0000: new disk, initializing [10076.908398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10080.283386] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10091.380686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10091.627732] Lustre: 285105:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10092.070087] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10097.748792] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10110.835868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10111.166808] Lustre: 286734:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10112.866400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10118.078372] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10128.939550] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10129.082767] Lustre: 287608:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10131.179429] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10131.186515] Lustre: Skipped 1 previous similar message [10131.271663] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10134.783636] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10143.619876] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10146.773312] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10146.778026] Lustre: Skipped 1 previous similar message [10161.766509] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 18:04:03 (1787609043) [10166.802562] Lustre: server umount lustre-MDT0000 complete [10167.289801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10167.311196] Lustre: Skipped 6 previous similar messages [10171.765620] LustreError: 285097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787609055 with bad export cookie 7722625213046590585 [10171.769977] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10171.775109] LustreError: 285097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10172.239458] Lustre: server umount lustre-MDT0001 complete [10186.248695] Lustre: server umount lustre-OST0000 complete [10188.234586] Lustre: 250291:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787609056/real 1787609056] req@ffff9ff788d7ce00 x1874442923133312/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787609072 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10189.650970] Lustre: server umount lustre-OST0001 complete [10197.685455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10198.261707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10198.265286] Lustre: Skipped 7 previous similar messages [10202.324725] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10210.258925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10215.137793] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10221.611640] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10227.232125] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10233.504283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10240.692137] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10243.560430] LustreError: 291684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10243.597819] LustreError: 291684:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [10247.275529] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10262.508583] loop1: detected capacity change from 0 to 204800000 [10262.789396] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10262.801176] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10262.837859] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10267.944238] loop1: detected capacity change from 0 to 204800000 [10274.109014] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10274.184396] LustreError: 289842:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs9191: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10274.193266] LustreError: 289842:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs9191 [10274.209710] LustreError: 289842:0:(mgs_handler.c:517:mgs_target_reg()) fs9191-MDT0001: Error checking target index: rc = -13 [10274.239686] LustreError: fs9191-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10274.250693] LustreError: 293572:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [10274.257709] LustreError: 293572:0:(tgt_mount.c:2132:server_put_super()) no obd fs9191-MDT0001 [10274.262796] LustreError: 293572:0:(tgt_mount.c:116:server_deregister_mount()) fs9191-MDT0001 not registered [10274.305623] Lustre: server umount fs9191-MDT0001 complete [10274.309292] LustreError: 293572:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10282.946144] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10283.051807] Lustre: Setting parameter fs9191-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs9191-MDT0001 [10283.110599] Lustre: srv-fs9191-MDT0001: No data found on store. Initialize space. [10283.117552] Lustre: Skipped 4 previous similar messages [10283.203558] Lustre: fs9191-MDT0001: new disk, initializing [10283.206737] Lustre: Skipped 3 previous similar messages [10283.300835] Lustre: srv-fs9191-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10283.310687] Lustre: 294039:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9191-MDT0001-mdtlov: can't create new llog: rc = -115 [10283.323489] LustreError: 294039:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9191-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10287.515993] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10294.378932] Lustre: server umount fs9191-MDT0001 complete [10307.729164] loop1: detected capacity change from 0 to 204800000 [10313.097500] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10313.485459] Lustre: 295607:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9191-MDT0001-mdtlov: can't create new llog: rc = -115 [10313.495844] LustreError: 295607:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9191-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10317.313857] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10320.378704] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10321.917934] Lustre: server umount fs9191-MDT0001 complete [10333.667255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10333.677566] LustreError: Skipped 1 previous similar message [10333.686604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10333.700227] Lustre: Skipped 2 previous similar messages [10333.714553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10333.721836] Lustre: Skipped 1 previous similar message [10337.192417] Lustre: server umount lustre-MDT0000 complete [10345.202808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10345.333163] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10349.321252] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10351.084987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10355.082661] loop1: detected capacity change from 0 to 204800000 [10359.312703] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10359.615256] Lustre: 298054:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9191-MDT0001-mdtlov: can't create new llog: rc = -115 [10359.630522] LustreError: 298054:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9191-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10363.057300] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10366.824821] Lustre: server umount fs9191-MDT0001 complete [10386.919060] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10400.223117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10400.536494] Lustre: server umount lustre-OST0000 complete [10410.571666] Lustre: server umount lustre-MDT0000 complete [10414.740047] LustreError: 289836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787609298 with bad export cookie 7722625213046593266 [10414.744586] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10414.746720] LustreError: 289836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10415.058727] Lustre: server umount lustre-MDT0001 complete [10427.615366] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [10430.533431] Key type lgssc unregistered [10430.834867] LNet: 300401:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10430.840565] LNetError: 300401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10430.857367] LNet: Removed LNI 192.168.206.103@tcp [10431.754144] Key type .llcrypt unregistered [10431.756936] Key type ._llcrypt unregistered [10450.142309] Key type ._llcrypt registered [10450.144608] Key type .llcrypt registered [10450.261383] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [10451.310095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10451.408620] alg: No test for adler32 (adler32-zlib) [10452.547171] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [10452.894291] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [10454.599222] Key type lgssc registered [10455.810711] Lustre: Echo OBD driver; http://www.lustre.org/ [10463.046224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10463.058488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10464.417119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10467.835611] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10474.981702] LustreError: 301673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10475.000218] LustreError: 301673:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10475.175339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10475.541152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10479.897618] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10485.961169] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10491.160523] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10496.713987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10496.986727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10501.102635] LustreError: 303502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10501.344737] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10506.217626] LustreError: 303504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10507.343351] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10509.292178] LustreError: 303504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10509.303157] LustreError: 303504:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10514.017900] LustreError: 303504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10524.225376] LustreError: 303502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10524.258634] LustreError: 303502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10528.025237] loop1: detected capacity change from 0 to 204800000 [10528.296970] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10528.312515] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10528.330424] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10534.325056] loop1: detected capacity change from 0 to 204800000 [10543.121602] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10543.474539] Lustre: fs11821-OST0000: new disk, initializing [10543.483118] Lustre: srv-fs11821-OST0000: No data found on store. Initialize space. [10543.496683] Lustre: 305406:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10543.721596] Lustre: fs11821-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10544.708165] LustreError: 303503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10544.723700] LustreError: 303503:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [10550.978757] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10559.669273] Lustre: server umount fs11821-OST0000 complete [10570.809351] loop1: detected capacity change from 0 to 204800000 [10575.535648] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10575.789471] Lustre: fs11821-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10580.544293] LustreError: 304171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10580.565116] LustreError: 304171:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [10581.847452] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10586.083442] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10589.338619] loop2: detected capacity change from 0 to 204800000 [10589.522534] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10589.536663] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10589.545484] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10591.213609] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10594.520397] loop2: detected capacity change from 0 to 204800000 [10599.512607] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10599.578292] LustreError: fs11821-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10599.587504] LustreError: fs11821-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10599.596294] LustreError: 308447:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [10599.603750] LustreError: 308447:0:(tgt_mount.c:2132:server_put_super()) no obd fs11821-OST0001 [10599.608266] LustreError: 308447:0:(tgt_mount.c:116:server_deregister_mount()) fs11821-OST0001 not registered [10599.634311] Lustre: server umount fs11821-OST0001 complete [10599.637136] LustreError: 308447:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10608.200445] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10608.361949] Lustre: fs11821-OST0001: new disk, initializing [10608.368849] Lustre: srv-fs11821-OST0001: No data found on store. Initialize space. [10608.382030] Lustre: 308999:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10608.478569] Lustre: fs11821-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10615.104300] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10623.196336] Lustre: server umount fs11821-OST0000 complete [10631.374469] Lustre: server umount fs11821-OST0001 complete [10642.865217] loop1: detected capacity change from 0 to 204800000 [10646.988288] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10647.099649] LustreError: 304171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10647.120584] LustreError: 304171:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [10647.155736] Lustre: fs11821-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10652.471907] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10659.077687] loop2: detected capacity change from 0 to 204800000 [10664.180174] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10671.404983] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10679.815210] Lustre: server umount fs11821-OST0000 complete [10689.042244] Lustre: server umount fs11821-OST0001 complete [10703.841464] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10703.847761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10703.873407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10705.891382] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10705.912367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10705.940109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10708.527518] Lustre: server umount lustre-OST0000 complete [10716.128381] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10716.145673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10716.172566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10716.182926] Lustre: Skipped 2 previous similar messages [10718.962076] Lustre: server umount lustre-MDT0000 complete [10723.291858] LustreError: 301654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787609607 with bad export cookie 3137284251300093489 [10723.295677] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10723.302466] LustreError: 301654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10723.672048] Lustre: server umount lustre-MDT0001 complete [10735.747392] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [10738.941470] Key type lgssc unregistered [10739.230776] LNet: 314925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10739.253283] LNetError: 314925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10739.275995] LNet: Removed LNI 192.168.206.103@tcp [10740.122159] Key type .llcrypt unregistered [10740.129084] Key type ._llcrypt unregistered [10770.498377] Key type ._llcrypt registered [10770.502043] Key type .llcrypt registered [10770.617690] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [10785.061893] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [10786.395963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10786.417701] alg: No test for adler32 (adler32-zlib) [10787.470791] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [10787.745775] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [10789.457512] Key type lgssc registered [10790.708758] Lustre: Echo OBD driver; http://www.lustre.org/ [10826.957832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10827.002978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10828.350403] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10828.452537] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10828.539349] Lustre: lustre-MDT0000: new disk, initializing [10828.642144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10828.661303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10833.750347] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10846.386671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10846.505670] Lustre: 318787:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10846.570072] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10846.574916] Lustre: Skipped 1 previous similar message [10846.670407] Lustre: lustre-MDT0001: new disk, initializing [10846.828877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10846.957406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10846.979409] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10853.281350] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10862.455577] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10868.101916] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10875.851470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10876.109614] Lustre: lustre-OST0000: new disk, initializing [10876.113641] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10876.120982] Lustre: 320754:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10876.215236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10882.282507] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10884.801074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10884.810776] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10884.924885] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10892.716545] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10894.871075] Lustre: Mounted lustre-client - version 2.17.57_80_g20f51fc [10895.732224] Lustre: Unmounted lustre-client [10910.194497] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10910.215687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10913.051133] Lustre: server umount lustre-OST0000 complete [10918.370871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10918.377631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10918.394322] Lustre: Skipped 1 previous similar message [10918.411096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10918.421747] Lustre: Skipped 1 previous similar message [10920.416128] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10920.417953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10920.436658] Lustre: Skipped 1 previous similar message [10923.391988] Lustre: server umount lustre-MDT0000 complete [10927.268321] LustreError: 318778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787609811 with bad export cookie 431275017887690318 [10927.270067] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10927.294815] LustreError: 318778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10927.930972] Lustre: server umount lustre-MDT0001 complete [10939.884260] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [10942.811434] Key type lgssc unregistered [10956.703521] LNet: 1 local NIs in recovery (showing 1): 192.168.206.103@tcp [10957.535709] LNet: 322916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10957.551903] LNetError: 322916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11024.111628] LNet: Removed LNI 192.168.206.103@tcp [11025.329718] Key type .llcrypt unregistered [11025.338286] Key type ._llcrypt unregistered [11047.992726] Key type ._llcrypt registered [11048.012128] Key type .llcrypt registered [11048.144532] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [11068.609875] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [11069.945330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11069.989993] alg: No test for adler32 (adler32-zlib) [11071.323447] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [11072.010243] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [11074.032372] Key type lgssc registered [11076.268154] Lustre: Echo OBD driver; http://www.lustre.org/ [11113.588546] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 18:19:55 (1787609995) [11120.476811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11120.501279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11121.779156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11121.821542] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11121.904650] Lustre: lustre-MDT0000: new disk, initializing [11121.997391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11122.030169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11126.316947] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11136.729906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11136.852474] Lustre: 326140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11136.926966] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11136.930682] Lustre: Skipped 1 previous similar message [11137.029741] Lustre: lustre-MDT0001: new disk, initializing [11137.133175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11137.195608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11137.214059] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11141.567351] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11151.849571] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11157.367835] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11164.103892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11164.492226] Lustre: lustre-OST0000: new disk, initializing [11164.503466] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11164.515115] Lustre: 328107:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11164.638399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11171.660263] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11172.413132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11172.441683] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11172.666114] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11182.187461] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11203.552018] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11203.552286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11203.555288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11203.563983] LustreError: Skipped 1 previous similar message [11203.615811] Lustre: Skipped 1 previous similar message [11205.508552] Lustre: server umount lustre-OST0000 complete [11213.793523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11213.809608] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11213.828720] Lustre: Skipped 1 previous similar message [11213.830626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11216.883636] Lustre: server umount lustre-MDT0000 complete [11220.841644] LustreError: 326132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787610104 with bad export cookie 9824515971497365578 [11220.842553] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11220.854469] LustreError: 326132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11221.306822] Lustre: server umount lustre-MDT0001 complete [11233.999137] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [11236.827847] Key type lgssc unregistered [11237.275632] LNet: 330019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11237.293934] LNetError: 330019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11237.321703] LNet: Removed LNI 192.168.206.103@tcp [11238.584487] Key type .llcrypt unregistered [11238.586715] Key type ._llcrypt unregistered [11263.256233] Key type ._llcrypt registered [11263.259558] Key type .llcrypt registered [11263.415655] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [11264.487702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11264.663435] alg: No test for adler32 (adler32-zlib) [11265.911731] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [11266.278241] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [11268.072084] Key type lgssc registered [11269.476397] Lustre: Echo OBD driver; http://www.lustre.org/ [11278.007679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11278.032361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11279.739676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11284.717393] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11290.081786] LustreError: 331296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11290.106854] LustreError: 331296:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11295.218355] LustreError: 331297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11295.771086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11296.394263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11302.183816] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11310.551809] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11317.812830] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11327.370419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11327.644631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11336.129408] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11343.931568] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11359.711099] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11360.950449] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11365.856585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11365.862767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11365.867109] Lustre: Skipped 2 previous similar messages [11365.875822] Lustre: Skipped 1 previous similar message [11366.491562] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.3@tcp (stopping) [11370.977698] LustreError: 331296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11371.016240] Lustre: server umount lustre-MDT0000 complete [11375.039077] LustreError: 331277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787610258 with bad export cookie 2111033420707140825 [11375.049691] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11375.062366] LustreError: 331277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11376.098739] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11376.101961] LustreError: 331291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11376.140278] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11376.164719] LustreError: 331291:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11380.132448] LustreError: 331292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11380.132653] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11380.157864] LustreError: 331292:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11380.172418] Lustre: Skipped 1 previous similar message [11384.235841] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11384.245216] Lustre: Skipped 1 previous similar message [11389.409306] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11389.412965] LustreError: 333787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11389.450823] LustreError: 333787:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11389.698955] Lustre: server umount lustre-MDT0001 complete [11393.569600] Lustre: server umount lustre-OST0000 complete [11402.358129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11406.864911] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11409.320541] Lustre: 335501:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.103@tcp [11411.303272] Lustre: server umount MGS complete [11418.988544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11419.527216] LustreError: 336073:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11419.594448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11423.911931] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11432.530656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11432.780239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11438.053738] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11448.859620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11449.263808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11457.005333] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11464.266501] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11466.100266] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11472.126553] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11473.757355] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11480.034771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11480.039917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11480.081788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11480.087256] Lustre: Skipped 4 previous similar messages [11480.817595] Lustre: server umount lustre-OST0000 complete [11485.156114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11485.178414] Lustre: Skipped 1 previous similar message [11491.038343] Lustre: server umount lustre-MDT0000 complete [11494.536023] LustreError: 336055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787610378 with bad export cookie 2111033420707141665 [11494.537367] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11494.555307] LustreError: 336055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11494.876895] Lustre: server umount lustre-MDT0001 complete [11506.464340] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [11509.261496] Key type lgssc unregistered [11509.577582] LNet: 339512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11509.591050] LNetError: 339512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11509.622860] LNet: Removed LNI 192.168.206.103@tcp [11510.615258] Key type .llcrypt unregistered [11510.619448] Key type ._llcrypt unregistered [11528.869789] Key type ._llcrypt registered [11528.876323] Key type .llcrypt registered [11529.042691] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [11544.809848] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [11546.031278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11546.108575] alg: No test for adler32 (adler32-zlib) [11547.159743] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [11547.402387] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [11549.119721] Key type lgssc registered [11550.104971] Lustre: Echo OBD driver; http://www.lustre.org/ [11588.784466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11588.868841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11590.235675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11590.262287] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11590.381971] Lustre: lustre-MDT0000: new disk, initializing [11590.503386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11590.533288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11595.070677] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11606.097865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11606.241376] Lustre: 342674:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11606.348738] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11606.356725] Lustre: Skipped 1 previous similar message [11606.484512] Lustre: lustre-MDT0001: new disk, initializing [11606.590882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11606.654049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11606.664593] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11611.152711] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11620.778190] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11626.526774] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11633.231958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11633.501700] Lustre: lustre-OST0000: new disk, initializing [11633.504516] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11633.515534] Lustre: 344644:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11633.604355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11639.418775] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11642.957718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11642.965578] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11643.105197] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11648.315164] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11652.813544] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11653.025362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11657.457769] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11657.658837] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11663.840578] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11663.851762] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11663.855052] LustreError: Skipped 1 previous similar message [11663.858080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11663.882579] Lustre: Skipped 1 previous similar message [11665.892756] Lustre: server umount lustre-OST0000 complete [11673.056306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11673.069426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11673.084582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11673.089863] Lustre: Skipped 1 previous similar message [11674.081762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11674.101937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11674.113024] Lustre: Skipped 1 previous similar message [11675.720951] Lustre: server umount lustre-MDT0000 complete [11679.230134] LustreError: 342667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787610563 with bad export cookie 1863705603449001787 [11679.231151] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11679.243684] LustreError: 342667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11679.627181] Lustre: server umount lustre-MDT0001 complete [11687.285798] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 18:29:29 (1787610569) [11702.409307] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [11711.356565] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [11742.803612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11743.035636] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11743.075924] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11743.179339] Lustre: lustre-MDT0000: new disk, initializing [11743.254192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11743.270481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11747.126270] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11757.185660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11757.321440] Lustre: 349400:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11757.409532] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11757.413557] Lustre: Skipped 1 previous similar message [11757.489422] Lustre: lustre-MDT0001: new disk, initializing [11757.589385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11757.628183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11757.643253] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11761.631941] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11770.230545] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11776.193876] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11783.154562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11783.329813] Lustre: lustre-OST0000: new disk, initializing [11783.333870] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11783.339476] Lustre: 351368:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11783.419595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11788.699771] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11790.361612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11790.370732] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11790.462566] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11796.902904] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11831.279042] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11831.280286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11831.298916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11831.304142] Lustre: Skipped 1 previous similar message [11831.316676] Lustre: Skipped 1 previous similar message [11833.845096] Lustre: server umount lustre-OST0000 complete [11839.463402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11839.477522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11839.507196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11845.790972] Lustre: server umount lustre-MDT0000 complete [11846.625362] LustreError: 349412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11846.639851] LustreError: 349412:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11849.299839] LustreError: 349391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787610733 with bad export cookie 1863705603449002669 [11849.310203] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11849.321635] LustreError: 349391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11849.588784] Lustre: server umount lustre-MDT0001 complete [11860.251727] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [11863.125576] Key type lgssc unregistered [11863.374817] LNet: 353522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11863.384840] LNetError: 353522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11863.400987] LNet: Removed LNI 192.168.206.103@tcp [11864.151297] Key type .llcrypt unregistered [11864.152967] Key type ._llcrypt unregistered [11937.248981] Key type ._llcrypt registered [11937.253826] Key type .llcrypt registered [11937.337726] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [11937.622917] Key type .llcrypt unregistered [11937.627861] Key type ._llcrypt unregistered [11997.848844] Key type ._llcrypt registered [11997.851769] Key type .llcrypt registered [11997.973070] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [11998.254307] Key type .llcrypt unregistered [11998.256585] Key type ._llcrypt unregistered [12024.745546] Key type ._llcrypt registered [12024.752289] Key type .llcrypt registered [12024.889995] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [12025.269326] Key type .llcrypt unregistered [12025.272725] Key type ._llcrypt unregistered [12053.623376] Key type ._llcrypt registered [12053.625275] Key type .llcrypt registered [12053.722549] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [12053.944787] Key type .llcrypt unregistered [12053.947429] Key type ._llcrypt unregistered [12117.545420] Key type ._llcrypt registered [12117.549276] Key type .llcrypt registered [12117.644433] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [12117.918522] Key type .llcrypt unregistered [12117.920192] Key type ._llcrypt unregistered [12141.216426] Key type ._llcrypt registered [12141.219367] Key type .llcrypt registered [12141.311171] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [12141.538428] Key type .llcrypt unregistered [12141.543203] Key type ._llcrypt unregistered [12166.585881] Key type ._llcrypt registered [12166.588872] Key type .llcrypt registered [12166.698145] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [12167.020346] Key type .llcrypt unregistered [12167.026369] Key type ._llcrypt unregistered [12193.367417] Key type ._llcrypt registered [12193.370992] Key type .llcrypt registered [12193.444276] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [12193.735448] Key type .llcrypt unregistered [12193.737708] Key type ._llcrypt unregistered [12217.686403] Key type ._llcrypt registered [12217.689292] Key type .llcrypt registered [12217.788930] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [12218.008285] Key type .llcrypt unregistered [12218.011645] Key type ._llcrypt unregistered [12243.530281] Key type ._llcrypt registered [12243.536760] Key type .llcrypt registered [12243.636103] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [12243.950288] Key type .llcrypt unregistered [12243.953515] Key type ._llcrypt unregistered [12294.481483] Key type ._llcrypt registered [12294.488549] Key type .llcrypt registered [12294.653893] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [12295.044449] Key type .llcrypt unregistered [12295.046049] Key type ._llcrypt unregistered [12321.447489] Key type ._llcrypt registered [12321.449253] Key type .llcrypt registered [12321.555359] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [12321.838791] Key type .llcrypt unregistered [12321.841159] Key type ._llcrypt unregistered [12349.010353] Key type ._llcrypt registered [12349.011895] Key type .llcrypt registered [12349.106455] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [12349.390422] Key type .llcrypt unregistered [12349.398320] Key type ._llcrypt unregistered [12392.951505] Key type ._llcrypt registered [12392.955678] Key type .llcrypt registered [12393.042970] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [12405.237763] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [12406.323981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12406.361675] alg: No test for adler32 (adler32-zlib) [12407.403681] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [12407.569783] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [12409.201087] Key type lgssc registered [12410.036923] Lustre: Echo OBD driver; http://www.lustre.org/ [12438.435787] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12438.455761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12439.674791] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12439.702761] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12439.771894] Lustre: tst250fs-MDT0000: new disk, initializing [12439.830854] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12439.847195] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12443.709745] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [12453.092911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12453.318444] Lustre: 373657:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12453.367929] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12453.371869] Lustre: Skipped 1 previous similar message [12453.441897] Lustre: tst250fs-MDT0001: new disk, initializing [12453.522082] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12453.573389] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12453.582545] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12457.564680] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [12462.110352] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12468.791181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12469.032862] Lustre: tst250fs-OST0000: Not available for connect from 0@lo (not set up) [12469.045465] Lustre: tst250fs-OST0000: new disk, initializing [12469.050675] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12469.059509] Lustre: 375292:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12469.174355] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12474.344828] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [12474.376079] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12474.391624] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12474.414525] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [12483.592616] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12483.765195] Lustre: tst250fs-OST0001: new disk, initializing [12483.768341] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12483.775631] Lustre: 376159:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12483.899322] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12486.501212] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12486.508220] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12486.553739] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12489.151272] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [12494.134196] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12496.496790] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [12496.504427] Lustre: Skipped 2 previous similar messages [12512.239918] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12512.248415] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12512.252091] Lustre: Skipped 1 previous similar message [12512.258819] Lustre: Skipped 3 previous similar messages [12517.347572] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12517.357419] Lustre: Skipped 3 previous similar messages [12522.467837] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12522.473755] Lustre: Skipped 3 previous similar messages [12526.047500] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12526.244212] Lustre: server umount tst250fs-MDT0000 complete [12529.746908] LustreError: 376158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787611413 with bad export cookie 6182778721990545614 [12529.747679] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12529.756088] LustreError: 376158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12530.059231] Lustre: server umount tst250fs-MDT0001 complete [12542.366779] Lustre: server umount tst250fs-OST0000 complete [12556.046466] Lustre: server umount tst250fs-OST0001 complete [12570.858366] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [12577.927343] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [12602.079367] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 18:44:44 (1787611484) [12603.388870] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12615.999994] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [12624.059743] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [12648.688198] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12361 sec ========= 18:45:30 (1787611530) [12650.192799] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:45:32 (1787611532) === [12653.229468] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:45:35 (1787611535) === [12685.146580] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [12687.748621] Key type lgssc unregistered [12688.025803] LNet: 385306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12688.030684] LNetError: 385306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12688.049278] LNet: Removed LNI 192.168.206.103@tcp [12688.867141] Key type .llcrypt unregistered [12688.875310] Key type ._llcrypt unregistered