[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 499597065 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003357] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.007856] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010129] LSM: Security Framework initializing [ 0.012036] Yama: becoming mindful. [ 0.013045] SELinux: Initializing. [ 0.013840] *** VALIDATE selinux *** [ 0.021670] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026559] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028107] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029115] *** VALIDATE tmpfs *** [ 0.031279] *** VALIDATE proc *** [ 0.032302] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.034295] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035144] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037029] Spectre V2 : User space: Vulnerable [ 0.038008] Speculative Store Bypass: Vulnerable [ 0.040865] debug: unmapping init [mem 0xffffffffbca59000-0xffffffffbca60fff] [ 0.042263] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043835] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044031] ... version: 2 [ 0.045021] ... bit width: 48 [ 0.046015] ... generic registers: 4 [ 0.047017] ... value mask: 0000ffffffffffff [ 0.048010] ... max period: 00007fffffffffff [ 0.049010] ... fixed-purpose events: 3 [ 0.050009] ... event mask: 000000070000000f [ 0.051276] rcu: Hierarchical SRCU implementation. [ 0.053534] smp: Bringing up secondary CPUs ... [ 0.054597] x86: Booting SMP configuration: [ 0.055022] .... node #0, CPUs: #1 #2 #3 [ 0.059729] smp: Brought up 1 node, 4 CPUs [ 0.061015] smpboot: Max logical packages: 1 [ 0.062014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.130031] node 0 deferred pages initialised in 65ms [ 0.135094] devtmpfs: initialized [ 0.137344] x86/mm: Memory block size: 128MB [ 0.143768] gcov: version magic: 0x41383552 [ 0.149285] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150079] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151180] pinctrl core: initialized pinctrl subsystem [ 0.153215] [ 0.154007] ************************************************************* [ 0.157409] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161013] ** ** [ 0.164011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.168013] ** ** [ 0.170011] ** This means that this kernel is built to expose internal ** [ 0.173013] ** IOMMU data structures, which may compromise security on ** [ 0.175010] ** your system. ** [ 0.178013] ** ** [ 0.180010] ** If you see this message and you are not debugging the ** [ 0.182015] ** kernel, report this immediately to your vendor! ** [ 0.184011] ** ** [ 0.186012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.189013] ************************************************************* [ 0.192072] NET: Registered protocol family 16 [ 0.194809] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.199063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.202299] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.207105] cpuidle: using governor menu [ 0.209000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.212828] PCI: Using configuration type 1 for base access [ 0.215126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.227100] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.230024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.234068] cryptd: max_cpu_qlen set to 1000 [ 0.236245] ACPI: Added _OSI(Module Device) [ 0.238014] ACPI: Added _OSI(Processor Device) [ 0.239012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.241011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.246514] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.253449] ACPI: Interpreter enabled [ 0.255057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.256011] ACPI: Using IOAPIC for interrupt routing [ 0.258100] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.262409] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.271845] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.274039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.277018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.280066] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.287695] acpiphp: Slot [2] registered [ 0.290098] acpiphp: Slot [5] registered [ 0.292211] acpiphp: Slot [6] registered [ 0.294141] acpiphp: Slot [7] registered [ 0.296126] acpiphp: Slot [8] registered [ 0.300093] acpiphp: Slot [9] registered [ 0.302123] acpiphp: Slot [10] registered [ 0.304114] acpiphp: Slot [3] registered [ 0.305094] acpiphp: Slot [4] registered [ 0.307082] acpiphp: Slot [11] registered [ 0.308093] acpiphp: Slot [12] registered [ 0.310099] acpiphp: Slot [13] registered [ 0.311096] acpiphp: Slot [14] registered [ 0.313093] acpiphp: Slot [15] registered [ 0.315114] acpiphp: Slot [16] registered [ 0.317116] acpiphp: Slot [17] registered [ 0.318169] acpiphp: Slot [18] registered [ 0.320085] acpiphp: Slot [19] registered [ 0.322102] acpiphp: Slot [20] registered [ 0.323128] acpiphp: Slot [21] registered [ 0.325116] acpiphp: Slot [22] registered [ 0.326088] acpiphp: Slot [23] registered [ 0.327110] acpiphp: Slot [24] registered [ 0.329085] acpiphp: Slot [25] registered [ 0.330096] acpiphp: Slot [26] registered [ 0.332116] acpiphp: Slot [27] registered [ 0.334101] acpiphp: Slot [28] registered [ 0.336097] acpiphp: Slot [29] registered [ 0.338125] acpiphp: Slot [30] registered [ 0.340100] acpiphp: Slot [31] registered [ 0.342062] PCI host bridge to bus 0000:00 [ 0.343016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.346020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.348019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.351024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.354142] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.358026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.360301] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.364205] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.368625] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.382012] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.386986] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.390016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.392015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.396017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.398601] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.402042] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.406055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.410026] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.417025] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.430027] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.436027] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.443758] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.450016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.457017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.491018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.501254] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.509024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.518026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.550025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.563069] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.571015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.579016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.598016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.608000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.616027] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.629024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.653023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.669075] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.678015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.687014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.708017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.718329] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.726028] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.734023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.751023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.767905] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.770494] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.773353] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.779360] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.782233] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.788097] iommu: Default domain type: Passthrough [ 0.790510] SCSI subsystem initialized [ 0.792200] ACPI: bus type USB registered [ 0.794143] usbcore: registered new interface driver usbfs [ 0.797120] usbcore: registered new interface driver hub [ 0.799108] usbcore: registered new device driver usb [ 0.801333] pps_core: LinuxPPS API ver. 1 registered [ 0.803019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.806088] PTP clock support registered [ 0.809207] EDAC MC: Ver: 3.0.0 [ 0.812035] PCI: Using ACPI for IRQ routing [ 0.815628] NetLabel: Initializing [ 0.818012] NetLabel: domain hash size = 128 [ 0.821023] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.823074] NetLabel: unlabeled traffic allowed by default [ 0.826079] vgaarb: loaded [ 0.827345] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.830025] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.837072] clocksource: Switched to clocksource kvm-clock [ 0.967571] VFS: Disk quotas dquot_6.6.0 [ 0.971976] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.974413] *** VALIDATE ramfs *** [ 0.975710] *** VALIDATE hugetlbfs *** [ 0.977268] pnp: PnP ACPI init [ 0.979608] pnp: PnP ACPI: found 6 devices [ 0.994906] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.998209] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.000323] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.002734] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.005306] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.008623] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.013387] NET: Registered protocol family 2 [ 1.015735] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.022938] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.026673] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.032573] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.036553] TCP: Hash tables configured (established 65536 bind 65536) [ 1.040000] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.042918] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.045947] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.049654] NET: Registered protocol family 1 [ 1.053381] RPC: Registered named UNIX socket transport module. [ 1.055614] RPC: Registered udp transport module. [ 1.057193] RPC: Registered tcp transport module. [ 1.058969] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.062075] NET: Registered protocol family 44 [ 1.064270] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.067207] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.069365] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.071729] PCI: CLS 0 bytes, default 64 [ 1.073444] Unpacking initramfs... [ 2.506197] debug: unmapping init [mem 0xffff9547bcc54000-0xffff9547bffbffff] [ 2.510089] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.512409] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.514800] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.071328] Initialise system trusted keyrings [ 3.073427] Key type blacklist registered [ 3.077460] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.088754] zbud: loaded [ 3.092389] *** VALIDATE nfs *** [ 3.093910] *** VALIDATE nfs4 *** [ 3.095623] pstore: using deflate compression [ 3.099657] Platform Keyring initialized [ 3.228366] NET: Registered protocol family 38 [ 3.230627] Key type asymmetric registered [ 3.232336] Asymmetric key parser 'x509' registered [ 3.234423] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.238147] io scheduler mq-deadline registered [ 3.240144] io scheduler kyber registered [ 3.241866] io scheduler bfq registered [ 3.244225] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.248198] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.251990] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.254756] ACPI: Power Button [PWRF] [ 3.262109] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.269787] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.290761] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.300503] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.332607] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.362336] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.427116] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.435643] Non-volatile memory driver v1.3 [ 3.437811] Linux agpgart interface v0.103 [ 3.474740] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.478990] vda: detected capacity change from 0 to 74698752 [ 3.495392] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.498921] vdb: detected capacity change from 0 to 1073741824 [ 3.514808] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.517361] vdc: detected capacity change from 0 to 2621440000 [ 3.533481] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.536455] vdd: detected capacity change from 0 to 2621440000 [ 3.555066] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.558178] vde: detected capacity change from 0 to 4294967296 [ 3.572431] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.575404] vdf: detected capacity change from 0 to 4294967296 [ 3.581559] libphy: Fixed MDIO Bus: probed [ 3.586806] usbcore: registered new interface driver usbserial_generic [ 3.588937] usbserial: USB Serial support registered for generic [ 3.590877] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.594566] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.596158] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.598201] mousedev: PS/2 mouse device common for all mice [ 3.600731] rtc_cmos 00:05: RTC can wake from S4 [ 3.603214] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.605472] rtc_cmos 00:05: registered as rtc0 [ 3.607724] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.610331] intel_pstate: CPU model not supported [ 3.612818] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.615226] hid: raw HID events driver (C) Jiri Kosina [ 3.618626] usbcore: registered new interface driver usbhid [ 3.620668] usbhid: USB HID core driver [ 3.620845] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.621979] drop_monitor: Initializing network drop monitor service [ 3.626930] Initializing XFRM netlink socket [ 3.628925] NET: Registered protocol family 10 [ 3.632257] Segment Routing with IPv6 [ 3.633834] NET: Registered protocol family 17 [ 3.636415] mpls_gso: MPLS GSO support [ 3.642218] RAS: Correctable Errors collector initialized. [ 3.644623] AVX version of gcm_enc/dec engaged. [ 3.646576] AES CTR mode by8 optimization enabled [ 3.737436] sched_clock: Marking stable (3737347688, 0)->(4638901797, -901554109) [ 3.741177] registered taskstats version 1 [ 3.744142] Loading compiled-in X.509 certificates [ 3.745797] zswap: loaded using pool lzo/zbud [ 3.772135] Key type big_key registered [ 3.787469] Key type encrypted registered [ 3.789390] ima: No TPM chip found, activating TPM-bypass! [ 3.791751] ima: Allocated hash algorithm: sha1 [ 3.793934] ima: No architecture policies found [ 3.797764] evm: Initialising EVM extended attributes: [ 3.799791] evm: security.selinux [ 3.801364] evm: security.ima [ 3.802751] evm: security.capability [ 3.804378] evm: HMAC attrs: 0x1 [ 3.808609] rtc_cmos 00:05: setting system clock to 2026-09-03 12:55:29 UTC (1788440129) [ 3.815597] debug: unmapping init [mem 0xffffffffbda03000-0xffffffffbdbfffff] [ 3.818571] debug: unmapping init [mem 0xffffffffbc782000-0xffffffffbca58fff] [ 3.829474] Write protecting the kernel read-only data: 28672k [ 3.832785] debug: unmapping init [mem 0xffffffffbae03000-0xffffffffbaffffff] [ 3.836083] debug: unmapping init [mem 0xffffffffbb714000-0xffffffffbb7fffff] [ 3.880855] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.890739] systemd[1]: Detected virtualization kvm. [ 3.892729] systemd[1]: Detected architecture x86-64. [ 3.894860] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.920362] systemd[1]: No hostname configured. [ 3.921902] systemd[1]: Set hostname to . [ 3.924127] random: systemd: uninitialized urandom read (16 bytes read) [ 3.926490] systemd[1]: Initializing machine ID from random generator. [ 3.964481] random: ln: uninitialized urandom read (6 bytes read) [ 4.062499] random: systemd: uninitialized urandom read (16 bytes read) [ 4.065857] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.070211] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.075473] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.732172] device-mapper: uevent: version 1.0.3 [ 4.735132] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.500014] random: fast init done [ 5.507760] virtio_net virtio0 ens2: renamed from eth0 [ 5.584284] scsi host0: ata_piix [ 5.609860] scsi host1: ata_piix [ 5.683468] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.685666] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.864484] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.322212] random: crng init done [ 10.327766] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 12.212901] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.683687] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.719755] SELinux: Disabled at runtime. [ 15.830790] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.844881] systemd[1]: Detected virtualization kvm. [ 15.848692] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.446220] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.453431] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.461625] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.473299] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.478560] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.494636] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.516287] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. [ 17.723533] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 18.954736] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.803838] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.105576] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.699885] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.736389] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 25.469660] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 26.260623] NFS: Registering the id_resolver key type [ 26.264371] Key type id_resolver registered [ 26.272027] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg614-server login: [ 44.550007] hrtimer: interrupt took 1523891 ns [ 89.119773] libcfs: loading out-of-tree module taints kernel. [ 89.192282] Key type ._llcrypt registered [ 89.198073] Key type .llcrypt registered [ 89.348964] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 108.726263] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 110.466097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 110.491111] alg: No test for adler32 (adler32-zlib) [ 112.262904] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 113.228330] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 115.271168] Key type lgssc registered [ 117.591382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 136.843289] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 182.914945] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 194.881623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 194.922895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 196.272449] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 196.363815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 196.497624] Lustre: lustre-MDT0000: new disk, initializing [ 196.627579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.648741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 200.987171] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.343279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 213.469826] Lustre: 6522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 213.517460] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 213.523564] Lustre: Skipped 1 previous similar message [ 213.675290] Lustre: lustre-MDT0001: new disk, initializing [ 213.757396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 213.793843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 213.808526] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 218.441616] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.989937] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 232.126307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 232.468231] Lustre: lustre-OST0000: new disk, initializing [ 232.474316] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 232.484598] Lustre: 8459:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 232.570835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 235.228412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 235.245395] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 235.362899] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 240.910046] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 256.429690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 256.559457] Lustre: lustre-OST0001: new disk, initializing [ 256.563960] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 256.571744] Lustre: 9533:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 256.656831] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 263.061740] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 264.714043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 264.739261] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 264.844328] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 275.402883] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 283.476576] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 290.828995] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing check_logdir /tmp/testlogs/ [ 297.146357] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing yml_node [ 301.716977] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 304.372146] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 306.791582] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 308.463244] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 09:00:31 EDT 2026 [ 326.457820] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 328.472978] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 346.591932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 346.593993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 346.624161] Lustre: Skipped 1 previous similar message [ 346.646106] Lustre: Skipped 3 previous similar messages [ 351.715630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 351.727336] Lustre: Skipped 3 previous similar messages [ 356.840408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 356.844221] Lustre: Skipped 3 previous similar messages [ 358.367356] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 358.768557] Lustre: server umount lustre-MDT0000 complete [ 362.982689] LustreError: 6514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440488 with bad export cookie 13081328215778570129 [ 362.985525] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 362.992399] LustreError: 6514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 367.072164] LustreError: 6528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 367.072819] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 367.078318] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 367.095496] LustreError: 6528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 367.112402] Lustre: Skipped 3 previous similar messages [ 372.196988] LustreError: 6530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.197190] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 372.228699] Lustre: Skipped 2 previous similar messages [ 373.217225] LustreError: 6529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 373.225258] LustreError: 6529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 377.311272] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 377.697811] Lustre: server umount lustre-MDT0001 complete [ 381.542452] Lustre: server umount lustre-OST0000 complete [ 385.715457] Lustre: server umount lustre-OST0001 complete [ 392.583861] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 402.069235] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 437.847192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 438.202250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 438.244454] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 438.385296] Lustre: lustre-MDT0000: new disk, initializing [ 438.495109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 438.518862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 443.848896] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 454.651216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 454.820218] Lustre: 15101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 454.887693] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 454.892506] Lustre: Skipped 1 previous similar message [ 454.979818] Lustre: lustre-MDT0001: new disk, initializing [ 455.068227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 455.123887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 455.138879] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 460.290206] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 470.141977] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 477.622958] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 485.875863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 486.314652] Lustre: lustre-OST0000: new disk, initializing [ 486.327097] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 486.333430] Lustre: 17069:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 486.450515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 488.031640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 488.040840] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 488.231786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 493.774558] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 504.914752] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 509.455487] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 509.633617] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 514.857603] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 515.068639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 519.142061] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 519.143344] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 519.151479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 519.151487] Lustre: Skipped 2 previous similar messages [ 519.154229] LustreError: Skipped 1 previous similar message [ 519.164862] Lustre: Skipped 1 previous similar message [ 523.193419] Lustre: server umount lustre-OST0000 complete [ 526.816396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 526.823269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 533.110785] Lustre: server umount lustre-MDT0000 complete [ 534.501031] LustreError: 15111:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.531965] LustreError: 15111:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 536.814879] LustreError: 15093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440662 with bad export cookie 13081328215778571669 [ 536.836144] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 537.230887] Lustre: server umount lustre-MDT0001 complete [ 543.455944] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 09:04:26 (1788440666) [ 563.074655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 563.831467] LustreError: 19945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.951338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 568.798453] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 573.410928] LustreError: 19946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.427097] LustreError: 19946:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 578.684885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 583.412898] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 591.555668] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 597.637363] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 599.807771] Lustre: Failing over lustre-MDT0000 [ 600.070555] Lustre: server umount lustre-MDT0000 complete [ 604.003281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 604.012873] LustreError: 20673:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 604.046269] Lustre: Skipped 2 previous similar messages [ 604.090696] LustreError: 20673:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 619.487667] Lustre: 3655:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440729/real 1788440729] req@ffff95481bae5500 x1875315511167616/t0(0) o400->MGC192.168.206.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788440745 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 619.549646] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 621.069467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 629.874464] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb58a36b453c6bb1b [ 629.893636] Lustre: MGC192.168.206.114@tcp: Connection restored to 0@lo (at 0@lo) [ 630.272321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 630.285370] Lustre: Skipped 1 previous similar message [ 630.375932] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 635.368113] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 635.379898] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 635.420861] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 635.461085] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 647.729804] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 653.797239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 653.815943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 653.820428] Lustre: Skipped 1 previous similar message [ 653.841954] Lustre: Skipped 5 previous similar messages [ 658.379897] Lustre: server umount lustre-MDT0000 complete [ 658.914189] LustreError: 19945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 658.942885] LustreError: 19945:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 662.650933] LustreError: 19926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440788 with bad export cookie 13081328215778573083 [ 662.670121] LustreError: 19926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 662.673877] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 662.982757] Lustre: server umount lustre-MDT0001 complete [ 672.857104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 678.086745] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 687.403588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 692.297505] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 699.716556] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 706.046890] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 708.612507] Lustre: Failing over lustre-MDT0000 [ 708.904780] Lustre: server umount lustre-MDT0000 complete [ 712.756324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 712.771967] Lustre: Skipped 1 previous similar message [ 722.914592] LustreError: 23284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 722.927728] LustreError: 23284:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 728.864109] Lustre: 3654:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440838/real 1788440838] req@ffff9547066a4700 x1875315511198336/t0(0) o400->MGC192.168.206.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788440854 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 728.925030] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 729.569605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 739.296095] LustreError: 3652:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff954707c6b100 x1875315511201024/t0(0) o250->MGC192.168.206.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 739.619667] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 743.854447] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 744.928673] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 744.934900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 744.942617] Lustre: Skipped 1 previous similar message [ 744.968796] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 752.768947] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 757.920739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 757.928897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 757.944307] Lustre: Skipped 1 previous similar message [ 757.959183] Lustre: Skipped 1 previous similar message [ 763.369770] Lustre: server umount lustre-MDT0000 complete [ 768.402623] LustreError: 23268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440894 with bad export cookie 13081328215778574028 [ 768.406828] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 768.419341] LustreError: 23268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 768.889342] Lustre: server umount lustre-MDT0001 complete [ 778.708043] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 09:08:21 (1788440901) [ 797.046664] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 806.398547] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 847.556230] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 859.609060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 859.975231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 860.004381] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 860.107884] Lustre: lustre-MDT0000: new disk, initializing [ 860.289785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 860.300258] Lustre: Skipped 3 previous similar messages [ 860.321547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 865.449655] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 877.225748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 877.488441] Lustre: 29670:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 877.898238] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 884.773536] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 897.051757] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 904.245192] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 912.405896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 912.681978] Lustre: 31643:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 914.609234] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 914.710501] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 914.870892] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 914.893582] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 919.667320] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 930.218685] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 936.839867] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 960.997612] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 961.005699] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 961.014375] LustreError: Skipped 1 previous similar message [ 961.020062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 961.083988] Lustre: Skipped 1 previous similar message [ 964.917176] Lustre: server umount lustre-OST0000 complete [ 970.210602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 975.833853] Lustre: server umount lustre-MDT0000 complete [ 976.355318] LustreError: 29681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 976.379676] LustreError: 29681:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 979.466945] LustreError: 29662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441105 with bad export cookie 13081328215778574434 [ 979.473485] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 979.480748] LustreError: 29662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 979.976494] Lustre: server umount lustre-MDT0001 complete [ 990.462204] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [ 993.453853] Key type lgssc unregistered [ 993.698637] LNet: 33625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 993.711458] LNetError: 33625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 994.727052] LNet: Removed LNI 192.168.206.114@tcp [ 995.648160] Key type .llcrypt unregistered [ 995.650390] Key type ._llcrypt unregistered [ 1020.343198] Key type ._llcrypt registered [ 1020.348465] Key type .llcrypt registered [ 1020.498326] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 1022.328577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1022.382747] alg: No test for adler32 (adler32-zlib) [ 1023.673947] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1024.032361] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 1025.743224] Key type lgssc registered [ 1027.245121] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1036.509253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1036.541776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1038.210092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1043.177612] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1047.534475] LustreError: 35040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1047.549710] LustreError: 35040:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1052.641326] LustreError: 35041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1053.516256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1053.821855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1057.735782] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1065.962866] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1073.387062] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1081.622580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1082.234614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1090.409696] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1098.633620] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1099.521563] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1113.740315] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 09:13:56 (1788441236) [ 1116.001155] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1118.083046] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1125.960695] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 09:14:08 (1788441248) [ 1135.267286] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 09:14:18 (1788441258) [ 1136.493164] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1143.812591] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1143.815550] Lustre: Skipped 2 previous similar messages [ 1145.350676] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 09:14:28 (1788441268) [ 1148.946279] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1148.950337] Lustre: Skipped 3 previous similar messages [ 1165.500460] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1165.504731] Lustre: Skipped 6 previous similar messages [ 1168.525473] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 09:14:51 (1788441291) [ 1171.177454] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1174.619541] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1182.920364] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 09:15:06 (1788441306) [ 1184.708340] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1184.712216] Lustre: Skipped 3 previous similar messages [ 1197.968795] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 09:15:21 (1788441321) [ 1216.480227] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1216.487386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1216.487849] LustreError: Skipped 1 previous similar message [ 1216.493158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1216.499796] Lustre: Skipped 1 previous similar message [ 1218.697396] Lustre: server umount lustre-OST0000 complete [ 1222.624261] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1222.629127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1222.643086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1222.651469] Lustre: Skipped 1 previous similar message [ 1226.723792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1226.725738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1228.253745] Lustre: server umount lustre-MDT0000 complete [ 1231.722990] LustreError: 39102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441357 with bad export cookie 6338696233108758049 [ 1231.724569] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1231.734296] LustreError: 39102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1232.042602] Lustre: server umount lustre-MDT0001 complete [ 1243.284249] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [ 1245.891618] Key type lgssc unregistered [ 1246.179289] LNet: 42892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1246.193482] LNetError: 42892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1246.220718] LNet: Removed LNI 192.168.206.114@tcp [ 1247.165922] Key type .llcrypt unregistered [ 1247.168988] Key type ._llcrypt unregistered [ 1266.004257] Key type ._llcrypt registered [ 1266.006817] Key type .llcrypt registered [ 1266.121854] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 1284.369709] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 1286.066548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1286.136116] alg: No test for adler32 (adler32-zlib) [ 1287.226197] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1287.480965] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 1289.199282] Key type lgssc registered [ 1290.483764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1325.134195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1325.163573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1326.484919] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1326.522116] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1326.594487] Lustre: lustre-MDT0000: new disk, initializing [ 1326.687621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1326.706978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1330.939756] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1339.797778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1339.929788] Lustre: 46045:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1339.984170] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1339.987264] Lustre: Skipped 1 previous similar message [ 1340.052430] Lustre: lustre-MDT0001: new disk, initializing [ 1340.151991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1340.185223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1340.205268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1344.272313] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1354.171868] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1361.169930] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1368.919864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1369.169612] Lustre: lustre-OST0000: new disk, initializing [ 1369.173501] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1369.193184] Lustre: 48014:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1369.330945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1376.567048] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1377.836188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1377.847930] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1377.964215] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1387.134502] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1392.385993] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1392.529990] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1397.337190] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1397.673483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1403.875524] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1403.880719] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1403.882511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1403.901237] Lustre: Skipped 1 previous similar message [ 1403.931270] LustreError: Skipped 1 previous similar message [ 1405.894331] Lustre: server umount lustre-OST0000 complete [ 1412.068047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1412.069841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1412.071870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1412.071878] Lustre: Skipped 1 previous similar message [ 1412.083502] Lustre: Skipped 1 previous similar message [ 1413.087839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1413.099192] Lustre: Skipped 2 previous similar messages [ 1415.803833] Lustre: server umount lustre-MDT0000 complete [ 1420.382261] LustreError: 46037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441546 with bad export cookie 15319635787890751263 [ 1420.382805] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1420.401857] LustreError: 46037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1420.889075] Lustre: server umount lustre-MDT0001 complete [ 1430.176974] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 09:19:13 (1788441553) [ 1437.611564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1438.082976] LustreError: 50476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1438.115350] LustreError: 50476:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1438.168532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1442.633846] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1443.302614] LustreError: 50477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1447.394872] LustreError: 50476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1450.182180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1450.515413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1454.811272] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1462.516511] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1468.914906] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1475.391799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1475.743206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1481.664493] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1490.021877] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1494.288583] LustreError: 53005:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1495.182496] Lustre: Modifying parameter general.timeout=1 in log params [ 1496.790431] Lustre: Modifying parameter general.timeout=3 in log params [ 1496.794531] Lustre: Skipped 1 previous similar message [ 1499.302112] Lustre: Modifying parameter general.timeout=6 in log params [ 1499.306279] Lustre: Skipped 2 previous similar messages [ 1503.576895] Lustre: Modifying parameter general.timeout=11 in log params [ 1503.585585] Lustre: Skipped 4 previous similar messages [ 1512.078382] Lustre: Modifying parameter general.timeout=22 in log params [ 1512.088536] Lustre: Skipped 10 previous similar messages [ 1528.701709] Lustre: Modifying parameter general.timeout=39 in log params [ 1528.708316] Lustre: Skipped 16 previous similar messages [ 1561.284360] Lustre: Modifying parameter general.timeout=76 in log params [ 1561.287538] Lustre: Skipped 36 previous similar messages [ 1623.005950] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 09:22:26 (1788441746) [ 1625.322207] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 in log params [ 1625.336479] Lustre: Skipped 55 previous similar messages [ 1626.477620] Lustre: 59947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1628.426692] Lustre: 60045:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1632.483716] Lustre: 60243:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1632.493456] Lustre: 60243:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1636.607435] Lustre: 60457:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1636.621570] Lustre: 60457:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1663.083418] Lustre: 61817:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1663.109288] LustreError: 61817:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1667.003526] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 09:23:10 (1788441790) [ 1678.824421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1678.832119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1678.845876] Lustre: Skipped 1 previous similar message [ 1683.945349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1683.955583] Lustre: Skipped 4 previous similar messages [ 1685.013509] Lustre: server umount lustre-MDT0000 complete [ 1688.358694] LustreError: 50456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441814 with bad export cookie 15319635787890752145 [ 1688.365789] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1688.378485] LustreError: 50456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1688.611531] Lustre: server umount lustre-MDT0001 complete [ 1701.501305] Lustre: server umount lustre-OST0000 complete [ 1724.264530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1724.577984] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1724.835390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1729.287314] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1738.380618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1738.509903] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1738.551233] Lustre: 64344:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1743.503168] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1751.321973] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1756.505837] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1763.647882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1763.819739] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1764.172488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1764.179165] Lustre: Skipped 1 previous similar message [ 1770.697455] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1778.328943] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1780.055233] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1780.071896] Lustre: Skipped 21 previous similar messages [ 1780.124392] Lustre: 66912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1786.860194] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1786.866155] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1786.892789] Lustre: Skipped 1 previous similar message [ 1786.905972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1792.492833] LustreError: 66189:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1792.693755] Lustre: server umount lustre-OST0000 complete [ 1797.600659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1797.630757] Lustre: Skipped 2 previous similar messages [ 1802.726860] LustreError: 64381:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1802.729652] Lustre: server umount lustre-MDT0000 complete [ 1802.741829] LustreError: 64381:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1806.327964] LustreError: 64336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441932 with bad export cookie 15319635787890755155 [ 1806.333847] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1806.342170] LustreError: 64336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1806.813133] Lustre: server umount lustre-MDT0001 complete [ 1817.262336] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [ 1819.758130] Key type lgssc unregistered [ 1820.118419] LNet: 68122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1820.132166] LNetError: 68122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1820.151613] LNet: Removed LNI 192.168.206.114@tcp [ 1820.932651] Key type .llcrypt unregistered [ 1820.934857] Key type ._llcrypt unregistered [ 1846.441207] Key type ._llcrypt registered [ 1846.443913] Key type .llcrypt registered [ 1846.574512] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 1847.815657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1847.926597] alg: No test for adler32 (adler32-zlib) [ 1848.975866] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1849.205957] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 1850.951198] Key type lgssc registered [ 1852.114653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1859.431443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1859.472418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1860.959069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1864.852939] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1871.333834] LustreError: 69538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1871.351392] LustreError: 69538:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1872.512423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1873.081272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1877.386138] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1883.367346] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1888.670694] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1895.501574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1895.962426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1901.380267] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1908.869722] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1916.405264] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1924.068846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1924.080986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1924.093995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1927.648556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1927.653744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1927.659191] Lustre: Skipped 1 previous similar message [ 1927.679773] Lustre: Skipped 1 previous similar message [ 1929.954186] Lustre: server umount lustre-MDT0000 complete [ 1932.773136] LustreError: 69534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1932.808716] LustreError: 69534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1934.071817] LustreError: 69518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442059 with bad export cookie 7765411150191922589 [ 1934.073855] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1934.098883] LustreError: 69518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1934.542119] Lustre: server umount lustre-MDT0001 complete [ 1949.752634] Lustre: server umount lustre-OST0000 complete [ 1975.426964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1975.799768] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1975.839931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1976.162232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1980.291589] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1987.497045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1987.563863] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1987.584658] Lustre: 74409:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1988.065424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1991.878107] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1999.240352] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2004.769876] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2011.953656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2012.221416] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2012.760304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2018.908509] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2026.814989] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2028.568745] Lustre: 76986:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2030.692903] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2039.778074] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2039.787431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2039.809677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2039.820340] Lustre: Skipped 1 previous similar message [ 2042.847852] Lustre: server umount lustre-OST0000 complete [ 2049.504779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2049.509352] LustreError: Skipped 1 previous similar message [ 2049.517283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2049.527530] Lustre: Skipped 1 previous similar message [ 2049.533149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2049.545551] Lustre: Skipped 2 previous similar messages [ 2052.932336] Lustre: server umount lustre-MDT0000 complete [ 2055.653612] LustreError: 74419:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2055.672866] LustreError: 74419:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2056.280970] LustreError: 76288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442181 with bad export cookie 7765411150191923422 [ 2056.288718] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2056.292489] LustreError: 76288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2056.524926] Lustre: server umount lustre-MDT0001 complete [ 2066.125938] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [ 2068.736977] Key type lgssc unregistered [ 2068.954459] LNet: 78199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2068.962487] LNetError: 78199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2068.977580] LNet: Removed LNI 192.168.206.114@tcp [ 2069.640653] Key type .llcrypt unregistered [ 2069.644184] Key type ._llcrypt unregistered [ 2091.412844] Key type ._llcrypt registered [ 2091.415686] Key type .llcrypt registered [ 2091.515380] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 2092.290592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2092.381383] alg: No test for adler32 (adler32-zlib) [ 2093.438833] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2093.604736] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 2095.256868] Key type lgssc registered [ 2096.104046] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2102.843335] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2102.891412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2104.436938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2108.273411] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2115.047121] LustreError: 79613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2115.072973] LustreError: 79613:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2116.259743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2116.748732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2120.663838] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2127.114979] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2131.934359] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2137.372790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2137.810628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2142.765225] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2149.233957] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2155.399199] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2156.108496] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2157.495343] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2157.500403] Lustre: Skipped 1 previous similar message [ 2159.543813] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2159.548786] Lustre: Skipped 2 previous similar messages [ 2163.982445] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2163.987448] Lustre: Skipped 5 previous similar messages [ 2172.221298] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2172.225646] Lustre: Skipped 9 previous similar messages [ 2188.508443] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2188.516537] Lustre: Skipped 19 previous similar messages [ 2220.912698] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2220.918831] Lustre: Skipped 37 previous similar messages [ 2246.827302] Lustre: 87713:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2249.204535] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 09:32:52 (1788442372) [ 2250.895875] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2252.937862] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 09:32:55 (1788442375) [ 2257.891029] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2257.907098] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2257.919151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2257.930051] Lustre: Skipped 1 previous similar message [ 2263.010139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2268.131742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2268.148203] Lustre: Skipped 2 previous similar messages [ 2271.199177] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2271.401934] Lustre: server umount lustre-OST0000 complete [ 2275.297283] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2275.306381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2275.323458] Lustre: Skipped 1 previous similar message [ 2275.338086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2278.374350] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2283.489144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2283.500235] Lustre: Skipped 3 previous similar messages [ 2289.120046] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2291.362588] Lustre: server umount lustre-MDT0000 complete [ 2293.731369] LustreError: 79613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2293.768309] LustreError: 79613:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2295.556808] LustreError: 79594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442421 with bad export cookie 8710772747212929038 [ 2295.570348] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2295.920699] Lustre: server umount lustre-MDT0001 complete [ 2306.831602] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [ 2309.515780] Key type lgssc unregistered [ 2309.813330] LNet: 88968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2309.819511] LNetError: 88968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2309.841588] LNet: Removed LNI 192.168.206.114@tcp [ 2310.728909] Key type .llcrypt unregistered [ 2310.734203] Key type ._llcrypt unregistered [ 2315.102115] Key type ._llcrypt registered [ 2315.103671] Key type .llcrypt registered [ 2315.231184] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2320.190650] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules [ 2321.279603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2321.310734] alg: No test for adler32 (adler32-zlib) [ 2322.417141] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2322.737380] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 2324.458354] Key type lgssc registered [ 2326.168757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2327.517300] LustreError: 90186:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2344.094532] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 2348.890797] LustreError: 90186:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2352.817834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2352.840576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2354.393690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2358.734657] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2364.897640] LustreError: 91201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2364.910489] LustreError: 91201:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2367.723239] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 09:34:50 (1788442490) [ 2370.025395] LustreError: 91200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2372.115665] LustreError: 91232:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2372.125315] LustreError: 91232:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2372.130981] LustreError: 91232:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 2372.480707] Lustre: server umount lustre-MDT0000 complete [ 2384.615773] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [ 2387.414867] Key type lgssc unregistered [ 2387.697482] LNet: 92729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2387.703428] LNetError: 92729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2387.724813] LNet: Removed LNI 192.168.206.114@tcp [ 2388.510923] Key type .llcrypt unregistered [ 2388.513216] Key type ._llcrypt unregistered [ 2405.040989] Key type ._llcrypt registered [ 2405.043476] Key type .llcrypt registered [ 2405.165941] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 2406.301664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2406.428697] alg: No test for adler32 (adler32-zlib) [ 2407.507229] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2407.852664] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 2409.519197] Key type lgssc registered [ 2410.848845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2418.834325] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2418.858154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2420.405893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2425.166978] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2429.923756] LustreError: 93947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2429.945767] LustreError: 93947:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2434.137336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2434.740053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2439.776781] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2447.224845] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2452.893854] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2458.666087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2459.348667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2465.334322] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2472.772603] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2521.162580] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 09:37:23 (1788442643) [ 2522.954518] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2524.990594] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 09:37:27 (1788442647) [ 2531.309600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2531.330907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2531.340618] Lustre: Skipped 1 previous similar message [ 2531.391918] Lustre: Skipped 2 previous similar messages [ 2535.896461] Lustre: server umount lustre-MDT0000 complete [ 2536.429545] LustreError: 95554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2536.453219] LustreError: 95554:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2540.085486] LustreError: 95800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442665 with bad export cookie 11531426262901201876 [ 2540.091558] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2540.096370] LustreError: 95800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2540.527207] Lustre: server umount lustre-MDT0001 complete [ 2554.601644] Lustre: server umount lustre-OST0000 complete [ 2566.693371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2567.148109] LustreError: 98237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2567.302135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2572.678660] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2572.776418] LustreError: 98238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2576.869107] LustreError: 98237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2582.129835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2583.192542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2588.032975] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2595.525641] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2601.549872] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2615.342982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2615.559281] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2615.571974] LustreError: 98226:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2615.605344] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2615.617895] LustreError: 100320:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2615.627058] LustreError: 100320:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2615.632418] LustreError: 100320:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2615.727266] Lustre: server umount lustre-OST0000 complete [ 2615.730175] LustreError: 100320:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2625.601042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2625.759557] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2625.767145] LustreError: 98227:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2625.790386] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2625.798727] LustreError: 100788:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2625.804395] LustreError: 100788:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2625.811496] LustreError: 100788:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2625.868532] Lustre: server umount lustre-OST0000 complete [ 2625.871519] LustreError: 100788:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2635.993861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2636.314992] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2636.321524] Lustre: Found index 0 for lustre-OST0000, updating log [ 2636.332333] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2636.434475] Lustre: lustre-OST0000: new disk, initializing [ 2636.438301] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2636.443108] Lustre: 101324:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2636.703151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2643.165332] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2646.600422] Lustre: Failing over lustre-OST0000 [ 2646.804851] Lustre: server umount lustre-OST0000 complete [ 2652.450350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2652.472304] Lustre: Skipped 1 previous similar message [ 2652.482623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2652.495045] Lustre: Skipped 1 previous similar message [ 2657.763587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2657.773779] Lustre: Skipped 1 previous similar message [ 2660.928795] Lustre: server umount lustre-MDT0000 complete [ 2662.880307] LustreError: 98234:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2662.898959] LustreError: 98234:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2664.416890] LustreError: 98218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442790 with bad export cookie 11531426262901204620 [ 2664.426082] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2664.429304] LustreError: 98218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2664.822748] Lustre: server umount lustre-MDT0001 complete [ 2674.043827] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 09:39:56 (1788442796) [ 2689.564734] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 2699.438147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2699.973989] LustreError: 103671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2700.131106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2704.727262] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2713.421373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2718.868260] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2721.800520] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2728.441282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2729.056531] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2735.561820] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2740.267915] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2740.278305] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2740.301354] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2744.123671] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2744.215565] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2744.332437] Lustre: lustre-OST0001: new disk, initializing [ 2744.336585] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2744.342146] Lustre: 105989:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2744.361046] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:06 [ 2744.388711] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2744.396609] Lustre: lustre-OST0000: Recovery over after 0:04, of 1 clients 1 recovered and 0 were evicted. [ 2744.397792] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2744.418714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2744.720777] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2744.725611] Lustre: Skipped 2 previous similar messages [ 2751.086759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2751.095299] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2751.189975] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2751.750753] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2760.107495] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2766.386619] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2771.937375] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2771.949661] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2771.974917] Lustre: Skipped 2 previous similar messages [ 2771.986517] Lustre: Skipped 2 previous similar messages [ 2776.991647] Lustre: server umount lustre-MDT0001 complete [ 2777.064751] LustreError: 103672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2777.091969] LustreError: 103672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2786.850474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2786.946988] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2786.953601] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2787.006629] Lustre: 103660:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2787.312148] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2787.320886] Lustre: Skipped 2 previous similar messages [ 2788.194302] LustreError: 107526:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2788.200400] LustreError: 107526:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2788.206758] LustreError: 103663:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2788.342905] LustreError: 107532:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.114@tcp: cfg command failed: rc = -17 [ 2788.354352] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2788.354352] [ 2792.443261] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2792.939834] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2801.231259] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 09:42:04 (1788442924) [ 2818.060714] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 2835.638298] Lustre: 109602:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2858.660718] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2862.170474] Lustre: 110738:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2940.898715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2940.917476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2940.947162] Lustre: Skipped 1 previous similar message [ 2940.953482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2940.959728] Lustre: Skipped 1 previous similar message [ 2947.400923] Lustre: server umount lustre-MDT0000 complete [ 2951.085971] LustreError: 103653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443076 with bad export cookie 11531426262901205614 [ 2951.088506] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2951.099329] LustreError: 103653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2951.706959] Lustre: server umount lustre-MDT0001 complete [ 2966.179827] Lustre: server umount lustre-OST0000 complete [ 2968.032476] Lustre: 93259:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443077/real 1788443077] req@ffff954839c2a300 x1875317917833344/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788443093 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2968.076333] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2968.089218] Lustre: Skipped 3 previous similar messages [ 2970.440407] Lustre: server umount lustre-OST0001 complete [ 2976.923895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2988.557470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2998.223346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3008.473058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3028.640696] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 3038.718478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3038.749553] Lustre: lustre-MDT0000: reset Object Index mappings [ 3039.247202] LustreError: 114689:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3039.256866] LustreError: 114689:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3039.368842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3039.375747] Lustre: Skipped 1 previous similar message [ 3044.041495] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 3052.502954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3052.529679] Lustre: lustre-MDT0001: reset Object Index mappings [ 3057.845520] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 3060.600321] Lustre: 115836:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3067.128494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3073.680596] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 3078.077291] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3081.708120] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3083.453488] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3083.456312] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3083.569526] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3088.167860] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 3095.951324] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3099.320277] Lustre: 117712:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3141.861373] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 09:47:45 (1788443265) [ 3170.796091] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3170.798310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3170.809675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3170.809685] Lustre: Skipped 8 previous similar messages [ 3170.817967] Lustre: Skipped 2 previous similar messages [ 3175.652293] Lustre: server umount lustre-MDT0000 complete [ 3175.911373] LustreError: 114684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3175.924558] LustreError: 114684:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3178.946291] LustreError: 117716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443304 with bad export cookie 11531426262901252717 [ 3178.961727] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3178.965055] LustreError: 117716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3185.452708] Lustre: server umount lustre-MDT0001 complete [ 3195.081862] Lustre: server umount lustre-OST0000 complete [ 3204.994335] Lustre: server umount lustre-OST0001 complete [ 3211.627483] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 3221.242174] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 3255.788207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3255.988516] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3256.020955] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3256.093695] Lustre: lustre-MDT0000: new disk, initializing [ 3256.141613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3256.148456] Lustre: Skipped 3 previous similar messages [ 3256.167088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3260.469619] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 3266.526877] Lustre: server umount lustre-MDT0000 complete [ 3276.639638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3276.879268] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3276.889600] Lustre: 121980:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3276.902979] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3281.151625] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 3285.584921] Lustre: server umount lustre-MDT0000 complete [ 3293.361941] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 09:50:16 (1788443416) [ 3294.719201] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3296.160996] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 09:50:19 (1788443419) [ 3313.197400] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 3324.009169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3329.230889] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 3337.156989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3337.293539] Lustre: 124029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3337.344253] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3337.348851] Lustre: Skipped 1 previous similar message [ 3337.433489] Lustre: lustre-MDT0001: new disk, initializing [ 3337.578882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3337.593062] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3341.923524] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 3346.072621] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3352.142720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3352.391139] Lustre: lustre-OST0000: new disk, initializing [ 3352.398550] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3352.405246] Lustre: 125606:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3353.920254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3353.927996] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3354.034867] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3358.266487] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 3367.532423] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3367.680122] Lustre: lustre-OST0001: new disk, initializing [ 3367.684860] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3367.691848] Lustre: 126480:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3369.263700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3369.271289] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3369.348273] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3373.284856] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 3381.726981] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3385.484458] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3761.362953] Lustre: 132889:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3761.369369] Lustre: 132889:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3761.375352] Lustre: 132889:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3761.382564] Lustre: 132889:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3761.390447] Lustre: 132889:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3761.397519] Lustre: 132889:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4013.241845] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:02:15 (1788444135) [ 4028.902832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4028.913832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4028.921928] Lustre: Skipped 3 previous similar messages [ 4028.926704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4028.931025] Lustre: Skipped 7 previous similar messages [ 4034.157766] Lustre: server umount lustre-MDT0000 complete [ 4035.043601] LustreError: 125598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4035.068329] LustreError: 125598:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4037.561811] LustreError: 124021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444163 with bad export cookie 11531426262901318636 [ 4037.571488] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4040.168346] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4040.171086] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4040.186176] Lustre: Skipped 4 previous similar messages [ 4040.207912] Lustre: Skipped 5 previous similar messages [ 4044.069588] Lustre: server umount lustre-MDT0001 complete [ 4054.456767] Lustre: server umount lustre-OST0000 complete [ 4063.990521] Lustre: server umount lustre-OST0001 complete [ 4070.480229] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 4079.367526] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 4117.448595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4117.765815] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4117.814578] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4117.965718] Lustre: lustre-MDT0000: new disk, initializing [ 4118.066776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4118.070200] Lustre: Skipped 5 previous similar messages [ 4118.083533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4122.597795] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 4133.329259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4133.441156] Lustre: 135947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4133.484575] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4133.488151] Lustre: Skipped 1 previous similar message [ 4133.538987] Lustre: lustre-MDT0001: new disk, initializing [ 4133.673042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4133.684054] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4139.034421] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 4149.537723] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4156.659695] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4164.563739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4164.801382] Lustre: lustre-OST0000: new disk, initializing [ 4164.804790] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4164.810306] Lustre: 137914:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4166.316723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4166.325825] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4166.480856] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4172.107527] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 4182.158564] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4190.599857] Lustre: lustre-MDD0000: changelog on [ 4193.479979] Lustre: lustre-MDD0001: changelog on [ 5919.675750] Lustre: lustre-MDD0001: changelog off [ 5921.180725] Lustre: 139370:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5921.873793] Lustre: lustre-MDD0000: changelog off [ 5927.906768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5927.913761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5927.921226] Lustre: Skipped 1 previous similar message [ 5927.924038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5927.939623] Lustre: Skipped 3 previous similar messages [ 5933.024584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5938.145918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5938.152845] Lustre: Skipped 2 previous similar messages [ 5941.216030] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5941.521657] Lustre: server umount lustre-OST0000 complete [ 5946.336702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5946.341282] LustreError: Skipped 1 previous similar message [ 5946.345272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5946.355691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5948.384839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5951.960611] Lustre: server umount lustre-MDT0000 complete [ 5953.511071] LustreError: 135957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5953.528149] LustreError: 135957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5955.527927] LustreError: 135940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446081 with bad export cookie 11531426262902098198 [ 5955.544779] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5955.551594] LustreError: 135940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5955.971786] Lustre: server umount lustre-MDT0001 complete [ 5966.499653] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [ 5968.901974] Key type lgssc unregistered [ 5969.160136] LNet: 147022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5969.177832] LNetError: 147022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5969.190332] LNet: Removed LNI 192.168.206.114@tcp [ 5970.088151] Key type .llcrypt unregistered [ 5970.092607] Key type ._llcrypt unregistered [ 5987.421793] Key type ._llcrypt registered [ 5987.424690] Key type .llcrypt registered [ 5987.520030] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 6000.538357] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 6001.332961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6001.355408] alg: No test for adler32 (adler32-zlib) [ 6002.454463] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6002.748785] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 6004.431184] Key type lgssc registered [ 6005.349525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6038.763406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6038.788965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6040.029908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6040.082411] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6040.188485] Lustre: lustre-MDT0000: new disk, initializing [ 6040.256205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6040.280233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6044.750353] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6055.117733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6055.236121] Lustre: 150203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6055.274696] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6055.278539] Lustre: Skipped 1 previous similar message [ 6055.347364] Lustre: lustre-MDT0001: new disk, initializing [ 6055.454859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6055.518030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6055.537211] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6060.216335] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6069.516955] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6075.019172] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6082.903419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6083.265306] Lustre: lustre-OST0000: new disk, initializing [ 6083.269610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6083.276551] Lustre: 152176:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6083.366952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6089.878294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6089.900503] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6090.036857] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6090.690540] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6101.865857] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6109.302609] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_module obdecho/obdecho [ 6111.427934] LustreError: 153181:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6111.432451] LustreError: 153181:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6113.236503] LustreError: 153277:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6126.049503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6126.051494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6126.075714] Lustre: Skipped 2 previous similar messages [ 6126.096838] Lustre: Skipped 2 previous similar messages [ 6130.889211] Lustre: server umount lustre-MDT0000 complete [ 6134.254334] LustreError: 150194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446259 with bad export cookie 6963280002067776930 [ 6134.259457] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6134.275184] LustreError: 150194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6134.631608] Lustre: server umount lustre-MDT0001 complete [ 6147.629880] Lustre: server umount lustre-OST0000 complete [ 6167.546053] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 6176.904520] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 6210.437283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6210.611534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6210.629802] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6210.685743] Lustre: lustre-MDT0000: new disk, initializing [ 6210.743095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6210.770677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6214.509346] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6223.369949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6223.490473] Lustre: 156994:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6223.545412] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6223.548957] Lustre: Skipped 1 previous similar message [ 6223.613897] Lustre: lustre-MDT0001: new disk, initializing [ 6223.721977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6223.772080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6223.792338] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6228.034711] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6236.477663] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6242.045402] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6248.510496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6248.813838] Lustre: lustre-OST0000: new disk, initializing [ 6248.819493] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6248.832302] Lustre: 158961:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6248.944580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6254.691534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6254.699503] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6254.743572] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6254.821571] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6264.027396] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6269.677128] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6269.840735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6275.183109] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6275.462975] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6280.672309] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6280.677147] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6280.681901] LustreError: Skipped 1 previous similar message [ 6280.687056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6280.700028] Lustre: Skipped 1 previous similar message [ 6280.735472] Lustre: Skipped 1 previous similar message [ 6284.167122] Lustre: server umount lustre-OST0000 complete [ 6290.400749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6290.421707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6290.462437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6294.767834] Lustre: server umount lustre-MDT0000 complete [ 6296.033461] LustreError: 158103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6296.053194] LustreError: 158103:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6298.416419] LustreError: 156985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446424 with bad export cookie 6963280002067777889 [ 6298.416807] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6298.432215] LustreError: 156985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6298.759501] Lustre: server umount lustre-MDT0001 complete [ 6302.665800] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 10:40:25 (1788446425) [ 6304.265704] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6306.358269] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 10:40:29 (1788446429) [ 6313.916614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6314.310229] LustreError: 161371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6314.398857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6318.697295] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6319.594604] LustreError: 161372:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6324.704224] LustreError: 161371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6325.891796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6330.502244] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6337.224894] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6341.962223] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6348.168185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6348.468756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6348.473819] Lustre: Skipped 1 previous similar message [ 6353.957183] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6360.412927] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6365.723492] LustreError: 163952:0:(ldlm_resource.c:1207:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200000bd0:0x1:0x0].0xcb9f90b0 (00000000bf1bda92) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 6366.170858] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.14@tcp (stopping) [ 6366.178994] Lustre: Skipped 1 previous similar message [ 6367.205915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6367.213363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6367.233668] Lustre: Skipped 1 previous similar message [ 6367.992697] Lustre: server umount lustre-MDT0000 complete [ 6370.275440] LustreError: 161372:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6370.308993] LustreError: 161372:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6371.296464] LustreError: 161353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446496 with bad export cookie 6963280002067778771 [ 6371.296739] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6371.306833] LustreError: 161353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6375.398773] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6375.414611] Lustre: Skipped 2 previous similar messages [ 6375.425101] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6375.429057] Lustre: Skipped 1 previous similar message [ 6377.662652] Lustre: server umount lustre-MDT0001 complete [ 6383.473685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6395.877032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6396.104589] LustreError: 165213:0:(llog.c:1655:llog_backup()) MGC192.168.206.114@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6396.110275] Lustre: 165213:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.114@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6396.928777] LustreError: 165218:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6396.944636] LustreError: 165218:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6416.364500] LustreError: 165219:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6416.377365] LustreError: 165219:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 6452.154361] LustreError: 165218:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6452.172095] LustreError: 165218:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 27 previous similar messages [ 6457.311509] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6457.314959] Lustre: 165213:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.114@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6498.322310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6502.380754] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6503.396559] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6509.560567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6516.664062] LustreError: 165220:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6516.685321] LustreError: 165220:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 44 previous similar messages [ 6520.812074] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6522.349665] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6526.857107] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6528.375373] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6533.713752] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6535.143161] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6553.070547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6553.073817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6553.098667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6553.117402] Lustre: Skipped 4 previous similar messages [ 6557.925225] Lustre: server umount lustre-MDT0000 complete [ 6561.839297] LustreError: 161352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446687 with bad export cookie 6963280002067785225 [ 6561.845477] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6561.851132] LustreError: 161352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6562.175662] Lustre: server umount lustre-MDT0001 complete [ 6575.850185] Lustre: server umount lustre-OST0000 complete [ 6597.157040] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 6607.442518] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 6640.948574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6641.159463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6641.193213] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6641.272459] Lustre: lustre-MDT0000: new disk, initializing [ 6641.342985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6641.349897] Lustre: Skipped 1 previous similar message [ 6641.369360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6645.503035] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6655.259365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6655.365985] Lustre: 170319:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6655.569437] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6659.630687] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6668.547733] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6673.170677] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6679.564562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6679.853477] Lustre: lustre-OST0000: new disk, initializing [ 6679.859517] Lustre: Skipped 1 previous similar message [ 6679.863792] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6679.871072] Lustre: Skipped 2 previous similar messages [ 6679.878212] Lustre: 172289:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6685.803571] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6689.849064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6689.858560] Lustre: Skipped 1 previous similar message [ 6689.861913] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6689.971376] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6694.493780] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6698.801705] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6699.029475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6703.022754] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6703.277866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6705.639436] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6705.639535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6705.646478] LustreError: Skipped 1 previous similar message [ 6705.647706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6705.656758] Lustre: Skipped 3 previous similar messages [ 6705.669569] Lustre: Skipped 1 previous similar message [ 6711.340767] Lustre: server umount lustre-OST0000 complete [ 6716.896947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6730.207586] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6730.348289] Lustre: server umount lustre-MDT0000 complete [ 6731.233468] LustreError: 171444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6731.266639] LustreError: 171444:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 6733.712778] LustreError: 172288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446859 with bad export cookie 6963280002067802060 [ 6733.714455] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6733.723512] LustreError: 172288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6734.112952] Lustre: server umount lustre-MDT0001 complete [ 6742.866916] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 10:47:45 (1788446865) [ 6749.202481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6753.302510] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6761.084507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6765.871632] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6773.111376] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6777.887276] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6783.948734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6790.298574] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6796.857427] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6804.801236] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 10:48:48 (1788446928) [ 6810.082174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6810.085058] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6810.108423] Lustre: Skipped 2 previous similar messages [ 6810.114650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6810.114658] Lustre: Skipped 7 previous similar messages [ 6814.075476] Lustre: server umount lustre-OST0000 complete [ 6832.095173] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6832.299771] Lustre: server umount lustre-MDT0000 complete [ 6836.112456] LustreError: 174681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446961 with bad export cookie 6963280002067802942 [ 6836.134025] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6836.576279] Lustre: server umount lustre-MDT0001 complete [ 6847.615854] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [ 6850.012738] Key type lgssc unregistered [ 6850.262215] LNet: 178427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6850.267975] LNetError: 178427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6850.296537] LNet: Removed LNI 192.168.206.114@tcp [ 6851.083432] Key type .llcrypt unregistered [ 6851.091080] Key type ._llcrypt unregistered [ 6867.343675] Key type ._llcrypt registered [ 6867.345825] Key type .llcrypt registered [ 6867.451441] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 6868.310254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6868.318886] alg: No test for adler32 (adler32-zlib) [ 6869.350799] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6869.605598] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 6871.263230] Key type lgssc registered [ 6872.410721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6880.044305] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6880.062500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6895.647396] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6900.767481] LustreError: 179683:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.114@tcp: failed processing log, type 4: rc = -110 [ 6905.953267] LustreError: 179683:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6905.978416] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6905.990620] LustreError: MGC192.168.206.114@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6906.005278] LustreError: 179683:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6906.013075] LustreError: 179683:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 6906.019531] LustreError: 179683:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 6906.125963] Lustre: server umount lustre-OST0000 complete [ 6906.136042] LustreError: 179683:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6913.894469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6915.404839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6919.814921] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6925.794145] LustreError: 180183:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6925.804606] LustreError: 180183:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6928.268573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6928.542166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6932.721678] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6939.334341] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6945.752265] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6952.009661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6952.214376] LustreError: 181998:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6952.226123] Lustre: 181998:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.114@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6952.367240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6957.570511] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6964.136149] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6969.824726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6969.829789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6969.840359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6970.851580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6970.855712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6970.863642] Lustre: Skipped 1 previous similar message [ 6970.877736] Lustre: Skipped 2 previous similar messages [ 6972.213627] Lustre: server umount lustre-MDT0000 complete [ 6975.434671] LustreError: 180163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447101 with bad export cookie 1164935777368024952 [ 6975.447533] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6975.448231] LustreError: 180163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6975.763285] Lustre: server umount lustre-MDT0001 complete [ 6989.952546] Lustre: server umount lustre-OST0000 complete [ 6998.752785] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 7008.727566] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 7040.253816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7040.428565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7040.463856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7040.537717] Lustre: lustre-MDT0000: new disk, initializing [ 7040.613680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7040.625752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7044.461487] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7054.117644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7054.280551] Lustre: 185525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7054.347754] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7054.351163] Lustre: Skipped 1 previous similar message [ 7054.411928] Lustre: lustre-MDT0001: new disk, initializing [ 7054.493638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7054.527948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7054.541216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7058.855312] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7066.484615] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7072.541613] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7080.101641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7080.373097] Lustre: lustre-OST0000: new disk, initializing [ 7080.386441] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7080.401991] Lustre: 187490:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7080.515844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7086.588942] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7087.148845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7087.167119] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7087.274832] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7094.303817] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7098.810678] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7099.040705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7102.776905] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7102.999593] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7108.072678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7108.075611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7108.078154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7108.096660] LustreError: Skipped 1 previous similar message [ 7108.132522] Lustre: Skipped 1 previous similar message [ 7110.964566] Lustre: server umount lustre-OST0000 complete [ 7116.262278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7116.271536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7116.288141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7116.294049] Lustre: Skipped 1 previous similar message [ 7123.424288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7123.428597] Lustre: Skipped 3 previous similar messages [ 7129.055200] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7129.369303] Lustre: server umount lustre-MDT0000 complete [ 7132.642145] LustreError: 185530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7132.665615] LustreError: 185530:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7132.965432] LustreError: 185517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447258 with bad export cookie 1164935777368025666 [ 7132.969809] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7132.975507] LustreError: 185517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7133.274305] Lustre: server umount lustre-MDT0001 complete [ 7140.897438] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 10:54:24 (1788447264) [ 7156.007847] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [ 7158.131798] Key type lgssc unregistered [ 7158.315332] LNet: 190437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7158.322775] LNetError: 190437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7159.337410] LNet: Removed LNI 192.168.206.114@tcp [ 7159.968165] Key type .llcrypt unregistered [ 7159.972697] Key type ._llcrypt unregistered [ 7174.520046] Key type ._llcrypt registered [ 7174.524653] Key type .llcrypt registered [ 7174.671499] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 7175.518208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7175.708131] alg: No test for adler32 (adler32-zlib) [ 7176.814309] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7177.071954] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 7178.719781] Key type lgssc registered [ 7179.899420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7187.003384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7187.035558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7188.533378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7193.024498] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7198.692406] LustreError: 191634:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7198.723494] LustreError: 191634:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7201.372073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7201.869838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7206.535467] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7214.434228] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7219.953960] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7228.071279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7228.343840] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7228.348504] LustreError: 193499:0:(llog.c:1696:llog_backup()) MGC192.168.206.114@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7228.362976] Lustre: 193499:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.114@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7228.455630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7234.834498] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7241.736299] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7243.744865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7243.757217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7243.760112] Lustre: Skipped 1 previous similar message [ 7243.781745] Lustre: Skipped 2 previous similar messages [ 7246.820329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7246.827688] Lustre: Skipped 2 previous similar messages [ 7249.956461] Lustre: server umount lustre-MDT0000 complete [ 7251.941092] LustreError: 193022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7251.967930] LustreError: 193022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7253.209702] LustreError: 191614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447378 with bad export cookie 7349942360148303388 [ 7253.237136] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7255.071337] LustreError: 192364:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002f6f9121 x1875322918169984/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7255.246790] Lustre: server umount lustre-MDT0001 complete [ 7268.809422] Lustre: server umount lustre-OST0000 complete [ 7279.183925] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 7289.611625] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 7324.290912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7324.531833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7324.562280] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7324.665960] Lustre: lustre-MDT0000: new disk, initializing [ 7324.726457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7324.754524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7328.308305] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7336.705361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7336.817383] Lustre: 197025:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7336.860333] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7336.863056] Lustre: Skipped 1 previous similar message [ 7336.865675] LustreError: 197807:0:(llog.c:1696:llog_backup()) MGC192.168.206.114@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7336.871143] LustreError: 197807:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7336.875463] Lustre: 197807:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.114@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7336.889578] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7336.893562] Lustre: Skipped 1 previous similar message [ 7336.951537] Lustre: lustre-MDT0001: new disk, initializing [ 7337.054858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7337.112801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7337.134178] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7340.498700] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7348.431557] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7353.308212] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7359.237526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7359.523396] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7359.529455] Lustre: Skipped 1 previous similar message [ 7359.531488] LustreError: 198968:0:(llog.c:1696:llog_backup()) MGC192.168.206.114@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7359.541633] LustreError: 198968:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7359.547522] Lustre: 198968:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.114@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7359.559204] Lustre: 198968:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7359.592575] Lustre: lustre-OST0000: new disk, initializing [ 7359.602177] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7359.609803] Lustre: 198992:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7359.739971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7365.380642] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7365.735083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7365.741388] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7365.872868] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7374.478627] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7379.308640] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7379.488428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7383.830171] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7383.984356] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7386.614715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7386.616096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7386.632091] Lustre: Skipped 1 previous similar message [ 7386.643231] LustreError: Skipped 1 previous similar message [ 7386.646786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7386.662762] Lustre: Skipped 1 previous similar message [ 7391.714412] Lustre: server umount lustre-OST0000 complete [ 7396.833026] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7396.842106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7401.711861] Lustre: server umount lustre-MDT0000 complete [ 7401.959979] LustreError: 197918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7401.982810] LustreError: 197918:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7405.254136] LustreError: 197017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447530 with bad export cookie 7349942360148304102 [ 7405.256165] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7405.263091] LustreError: 197017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7405.630556] Lustre: server umount lustre-MDT0001 complete [ 7415.366985] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 10:58:58 (1788447538) [ 7430.926623] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 7439.754616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7440.028269] LustreError: 202117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7440.091764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7444.375475] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7445.472444] LustreError: 202118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7453.008807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7457.804208] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7460.911801] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7467.445392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7473.386737] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7481.851716] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7482.017838] Lustre: lustre-OST0001: new disk, initializing [ 7482.020870] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7482.027698] Lustre: 204422:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7482.124540] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7482.138683] Lustre: Skipped 2 previous similar messages [ 7488.010700] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7489.129337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7489.146195] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7489.246557] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7498.188726] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7504.498738] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7506.300871] Lustre: DEBUG MARKER: ADD OST3 [ 7511.118109] loop: module loaded [ 7511.229376] loop0: detected capacity change from 0 to 204800000 [ 7511.492158] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7511.503617] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7511.522401] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7517.122113] loop0: detected capacity change from 0 to 204800000 [ 7518.318261] Lustre: DEBUG MARKER: STOP OST3 [ 7523.317291] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7523.537239] Lustre: lustre-OST0003: new disk, initializing [ 7523.544319] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7523.556989] Lustre: 206417:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7525.257888] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7525.431955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7525.459033] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7525.510890] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7527.131481] Lustre: Failing over lustre-OST0003 [ 7527.237381] Lustre: server umount lustre-OST0003 complete [ 7527.273287] LustreError: 206609:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7529.533792] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7529.953370] LustreError: 203604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7529.969778] LustreError: 203604:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7530.481099] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7530.484221] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7530.492182] Lustre: Skipped 1 previous similar message [ 7537.684398] Lustre: DEBUG MARKER: START OST3 again [ 7543.110927] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7543.389620] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7544.512162] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7544.729729] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7544.734338] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7550.112522] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7570.471572] Lustre: Failing over lustre-OST0003 [ 7570.570935] Lustre: server umount lustre-OST0003 complete [ 7574.002653] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7574.018026] Lustre: Skipped 1 previous similar message [ 7574.025128] LustreError: 207865:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7574.044763] LustreError: 207865:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 7589.349835] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7589.353287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7589.362890] Lustre: Skipped 3 previous similar messages [ 7589.374603] Lustre: Skipped 4 previous similar messages [ 7593.263948] Lustre: server umount lustre-MDT0000 complete [ 7596.511760] LustreError: 202097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447722 with bad export cookie 7349942360148304991 [ 7596.514837] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7596.528948] LustreError: 202097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7599.596827] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7599.606615] Lustre: Skipped 1 previous similar message [ 7602.954260] Lustre: server umount lustre-MDT0001 complete [ 7606.810183] Lustre: server umount lustre-OST0000 complete [ 7610.108848] Lustre: server umount lustre-OST0001 complete [ 7615.899271] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 7622.804355] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 7659.390498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7659.576589] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7659.606637] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7659.697289] Lustre: lustre-MDT0000: new disk, initializing [ 7659.862924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7659.876494] Lustre: Skipped 2 previous similar messages [ 7659.901281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7664.579316] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7675.122245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7675.319189] Lustre: 211895:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7675.894886] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7682.133706] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7692.915963] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7699.128902] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7705.420676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7705.654818] Lustre: lustre-OST0000: new disk, initializing [ 7705.658466] Lustre: Skipped 1 previous similar message [ 7705.664428] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7705.668475] Lustre: Skipped 2 previous similar messages [ 7705.674588] Lustre: 213862:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7707.279145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7707.292910] Lustre: Skipped 1 previous similar message [ 7707.306257] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7707.403917] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7712.025201] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7721.541482] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7726.254403] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7726.482824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7731.179801] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7731.367133] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7733.217808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7733.225404] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7733.239439] Lustre: Skipped 3 previous similar messages [ 7733.253486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7733.253494] Lustre: Skipped 2 previous similar messages [ 7739.282072] Lustre: server umount lustre-OST0000 complete [ 7749.240373] Lustre: server umount lustre-MDT0000 complete [ 7752.731750] LustreError: 211888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447878 with bad export cookie 7349942360148307287 [ 7752.735323] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7752.746810] LustreError: 211888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7753.172318] Lustre: server umount lustre-MDT0001 complete [ 7756.828466] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:04:39 (1788447879) [ 7773.026608] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 7783.575995] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 7816.478508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7816.720660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7816.767301] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7816.908631] Lustre: lustre-MDT0000: new disk, initializing [ 7817.021807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7817.034264] Lustre: Skipped 2 previous similar messages [ 7817.070583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7821.202578] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7831.027304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7831.141785] Lustre: 218470:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7831.418470] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7836.019333] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7845.053546] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7850.684261] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7857.405816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7857.799466] Lustre: 220443:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7859.559728] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7864.108348] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7874.228172] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7879.858753] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7880.073953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7884.424876] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7884.718364] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7890.402471] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7890.405082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7890.406040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7890.406046] Lustre: Skipped 7 previous similar messages [ 7890.425173] Lustre: Skipped 3 previous similar messages [ 7892.718602] Lustre: server umount lustre-OST0000 complete [ 7898.081581] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7898.101304] LustreError: Skipped 1 previous similar message [ 7911.394071] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7911.934417] Lustre: server umount lustre-MDT0000 complete [ 7916.004217] LustreError: 218477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7916.024175] LustreError: 218477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 7916.353533] LustreError: 218463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448042 with bad export cookie 7349942360148308169 [ 7916.358041] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7916.363061] LustreError: 218463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7916.797924] Lustre: server umount lustre-MDT0001 complete [ 7927.275356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7932.943473] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7940.610503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7945.232476] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7952.800163] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7958.051546] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7965.424449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7973.941879] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7984.345523] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7996.901357] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7996.920080] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7996.927511] LustreError: Skipped 1 previous similar message [ 7996.934219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7996.943566] Lustre: Skipped 3 previous similar messages [ 7996.988959] Lustre: Skipped 9 previous similar messages [ 8000.418278] Lustre: server umount lustre-OST0000 complete [ 8018.399105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8018.724609] Lustre: server umount lustre-MDT0000 complete [ 8022.502840] LustreError: 222663:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8022.524290] LustreError: 222663:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 8023.600881] LustreError: 222643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448149 with bad export cookie 7349942360148309051 [ 8023.622087] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8023.628913] LustreError: 222643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8024.313371] Lustre: server umount lustre-MDT0001 complete [ 8121.449424] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [ 8124.252533] Key type lgssc unregistered [ 8124.546885] LNet: 226329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8124.553784] LNetError: 226329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8125.612327] LNet: Removed LNI 192.168.206.114@tcp [ 8126.413325] Key type .llcrypt unregistered [ 8126.417714] Key type ._llcrypt unregistered [ 8151.173482] Key type ._llcrypt registered [ 8151.176818] Key type .llcrypt registered [ 8151.278830] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 8166.467586] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 8167.728773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8167.850183] alg: No test for adler32 (adler32-zlib) [ 8169.061165] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8169.370319] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 8171.095469] Key type lgssc registered [ 8172.464985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8207.367858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8207.424148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8208.759795] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8208.800340] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8208.878264] Lustre: lustre-MDT0000: new disk, initializing [ 8208.932108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8208.948764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8213.930137] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8223.555714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8223.708677] Lustre: 229680:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8223.768467] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8223.774197] Lustre: Skipped 1 previous similar message [ 8223.893628] Lustre: lustre-MDT0001: new disk, initializing [ 8224.034576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8224.091279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8224.110770] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8229.063511] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8239.075113] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8245.342403] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8251.984733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8252.285802] Lustre: lustre-OST0000: new disk, initializing [ 8252.291156] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8252.295517] Lustre: 231652:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8252.375164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8258.898228] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8261.766702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8261.773938] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8261.892312] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8268.309817] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8274.010550] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8274.286892] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8279.945525] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8280.256379] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8287.712538] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8287.724982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8287.730671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8287.750314] LustreError: Skipped 1 previous similar message [ 8289.290190] Lustre: server umount lustre-OST0000 complete [ 8295.907771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8295.919791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8295.932150] Lustre: Skipped 1 previous similar message [ 8295.940314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8295.942687] Lustre: Skipped 1 previous similar message [ 8297.954445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8297.968128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8297.983436] Lustre: Skipped 1 previous similar message [ 8299.845800] Lustre: server umount lustre-MDT0000 complete [ 8303.827969] LustreError: 229674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448429 with bad export cookie 15928418049965881819 [ 8303.836342] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8303.851092] LustreError: 229674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8304.470383] Lustre: server umount lustre-MDT0001 complete [ 8324.377568] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 8335.837218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8336.124311] LustreError: 234588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8336.150735] LustreError: 234588:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8336.193375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8340.208587] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8341.495014] LustreError: 234589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8345.636426] LustreError: 234588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8347.936888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8348.453330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8352.704207] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8355.346564] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8361.147447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8361.593806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8367.442486] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8374.936337] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8375.095350] Lustre: lustre-OST0001: new disk, initializing [ 8375.098668] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8375.104582] Lustre: 236895:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8380.793403] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8381.498054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8381.515578] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8381.646208] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8391.867336] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8396.961847] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8408.395877] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 11:15:31 (1788448531) [ 8412.646525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8412.647944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8412.661264] Lustre: Skipped 2 previous similar messages [ 8412.674648] Lustre: Skipped 3 previous similar messages [ 8416.584632] Lustre: server umount lustre-MDT0000 complete [ 8417.766344] LustreError: 237422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8417.792626] LustreError: 237422:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8419.706411] LustreError: 235323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448545 with bad export cookie 15928418049965882694 [ 8419.709614] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8419.715266] LustreError: 235323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8419.980386] Lustre: server umount lustre-MDT0001 complete [ 8433.756568] Lustre: server umount lustre-OST0000 complete [ 8448.059672] Lustre: server umount lustre-OST0001 complete [ 8454.347915] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 8463.261847] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 8502.290947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8502.752034] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8502.791920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8502.916756] Lustre: lustre-MDT0000: new disk, initializing [ 8502.965691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8502.970303] Lustre: Skipped 1 previous similar message [ 8502.979931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8507.754038] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8520.809443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8520.990155] Lustre: 240837:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8521.047633] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8521.052295] Lustre: Skipped 1 previous similar message [ 8521.144278] Lustre: lustre-MDT0001: new disk, initializing [ 8521.370911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8521.384126] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8525.628925] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8535.978655] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8541.728310] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8549.480114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8549.664654] Lustre: lustre-OST0000: new disk, initializing [ 8549.667564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8549.674739] Lustre: 242808:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8549.750936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8549.761609] Lustre: Skipped 1 previous similar message [ 8551.339593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8551.349299] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8551.423369] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8555.271087] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8566.110535] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8571.992921] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8572.289895] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8577.723186] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8578.077907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8582.112564] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8582.114457] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8582.131095] LustreError: Skipped 1 previous similar message [ 8582.146944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8582.154382] Lustre: Skipped 2 previous similar messages [ 8582.171083] Lustre: Skipped 1 previous similar message [ 8586.568941] Lustre: server umount lustre-OST0000 complete [ 8592.356443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8592.360313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8592.373834] Lustre: Skipped 1 previous similar message [ 8592.389308] Lustre: Skipped 1 previous similar message [ 8597.051145] Lustre: server umount lustre-MDT0000 complete [ 8597.476209] LustreError: 240846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8597.500665] LustreError: 240846:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8600.598468] LustreError: 240829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448726 with bad export cookie 15928418049965884836 [ 8600.599964] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8600.607631] LustreError: 240829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8600.922399] Lustre: server umount lustre-MDT0001 complete [ 8608.454124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8608.815512] LustreError: 245026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8613.198271] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8621.530524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8621.865591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8621.875325] Lustre: Skipped 1 previous similar message [ 8626.334726] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8633.446517] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8638.473343] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8645.101365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8651.093858] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8659.707520] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8807.394982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8807.402163] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8807.421217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8820.191172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8820.392966] Lustre: server umount lustre-OST0000 complete [ 8831.332712] Lustre: server umount lustre-MDT0000 complete [ 8835.193887] LustreError: 245006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448960 with bad export cookie 15928418049965885718 [ 8835.205847] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8835.206483] LustreError: 245006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8835.694790] Lustre: server umount lustre-MDT0001 complete [ 8928.957667] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [ 8931.938360] Key type lgssc unregistered [ 8932.254533] LNet: 248636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8932.265769] LNetError: 248636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8933.292218] LNet: Removed LNI 192.168.206.114@tcp [ 8934.253222] Key type .llcrypt unregistered [ 8934.258449] Key type ._llcrypt unregistered [ 8959.880661] Key type ._llcrypt registered [ 8959.886167] Key type .llcrypt registered [ 8960.021265] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 8975.348243] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 8976.674559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8976.749882] alg: No test for adler32 (adler32-zlib) [ 8977.754286] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8977.998532] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 8979.752927] Key type lgssc registered [ 8981.087580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9027.606594] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 9037.591913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9037.653796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9038.951580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9039.005643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9039.130922] Lustre: lustre-MDT0000: new disk, initializing [ 9039.200627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9039.215861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9043.341066] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9052.975994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9053.121028] Lustre: 252727:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9053.175507] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9053.181191] Lustre: Skipped 1 previous similar message [ 9053.277563] Lustre: lustre-MDT0001: new disk, initializing [ 9053.375890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9053.433494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9053.445433] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9057.244911] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9061.570748] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9067.480141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9067.709944] Lustre: lustre-OST0000: new disk, initializing [ 9067.714132] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9067.719826] Lustre: 254359:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9067.840725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9072.192975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9072.209540] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9072.290029] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9072.907586] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9082.071681] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9082.225707] Lustre: lustre-OST0001: new disk, initializing [ 9082.237294] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9082.248832] Lustre: 255229:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9082.399996] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9088.395759] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9092.219217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9092.229679] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9092.359926] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9099.670763] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9105.207541] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9118.182594] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9118.185551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9118.196568] Lustre: Skipped 3 previous similar messages [ 9118.209329] Lustre: Skipped 3 previous similar messages [ 9122.528890] Lustre: server umount lustre-MDT0000 complete [ 9125.725174] LustreError: 252719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449251 with bad export cookie 16795476323767683138 [ 9125.734816] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9125.737311] LustreError: 252719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9126.099841] Lustre: server umount lustre-MDT0001 complete [ 9138.622479] Lustre: server umount lustre-OST0000 complete [ 9151.551960] Lustre: server umount lustre-OST0001 complete [ 9165.302164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9176.776981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9185.705711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9194.532981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9211.691164] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 9219.135904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9219.160431] Lustre: lustre-MDT0000: reset Object Index mappings [ 9219.435761] LustreError: 260237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9219.452448] LustreError: 260237:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9219.477913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9223.034916] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9224.673783] LustreError: 260238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9229.448272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9229.481151] Lustre: lustre-MDT0001: reset Object Index mappings [ 9229.835423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9233.373963] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9235.865611] Lustre: 261379:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9241.659321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9246.063103] LustreError: 261732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9246.076453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9247.048289] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9251.298360] LustreError: 261732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9254.337964] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9254.592835] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9254.597272] Lustre: Skipped 1 previous similar message [ 9259.997214] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9260.024564] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9268.530584] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9272.435861] Lustre: 263248:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9279.048042] Lustre: server umount lustre-MDT0000 complete [ 9280.993809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9281.003955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9281.033555] LustreError: 260237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9281.061781] LustreError: 260237:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9283.227684] LustreError: 260218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449408 with bad export cookie 16795476323767689928 [ 9283.248224] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9283.255529] LustreError: 260218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9283.558793] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9283.584540] Lustre: Skipped 2 previous similar messages [ 9283.906239] Lustre: server umount lustre-MDT0001 complete [ 9290.002543] Lustre: server umount lustre-OST0000 complete [ 9294.145691] Lustre: server umount lustre-OST0001 complete [ 9314.087896] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 11:30:36 (1788449436) [ 9330.865302] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 9339.517658] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 9380.187683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9380.432612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9380.463643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9380.553193] Lustre: lustre-MDT0000: new disk, initializing [ 9380.628729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9380.655062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9384.790351] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9393.822355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9393.956695] Lustre: 267500:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9394.057190] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9394.066111] Lustre: Skipped 1 previous similar message [ 9394.196158] Lustre: lustre-MDT0001: new disk, initializing [ 9394.354986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9394.372820] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9398.464082] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9407.954338] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9414.017830] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9420.113837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9420.433645] Lustre: lustre-OST0000: new disk, initializing [ 9420.439535] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9420.460111] Lustre: 269470:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9422.168734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9422.184371] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9422.281199] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9426.279740] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9437.735710] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9442.655940] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9442.859361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9447.180477] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9447.477663] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9453.025554] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9453.026333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9453.036069] LustreError: Skipped 1 previous similar message [ 9453.051288] Lustre: Skipped 1 previous similar message [ 9453.053637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9455.528590] Lustre: server umount lustre-OST0000 complete [ 9461.219336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9461.231509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9461.241576] Lustre: Skipped 1 previous similar message [ 9461.245788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9461.250661] Lustre: Skipped 1 previous similar message [ 9463.268087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9463.275610] Lustre: Skipped 1 previous similar message [ 9465.990753] Lustre: server umount lustre-MDT0000 complete [ 9468.392838] LustreError: 268872:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9468.403285] LustreError: 268872:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9469.861623] LustreError: 267494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449595 with bad export cookie 16795476323767692021 [ 9469.869476] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9469.882465] LustreError: 267494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9470.468720] Lustre: server umount lustre-MDT0001 complete [ 9494.656636] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 9504.561348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9504.891217] LustreError: 272418:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9504.932192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9504.940739] Lustre: Skipped 2 previous similar messages [ 9508.648402] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9516.790468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9521.252913] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9523.728226] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9529.419531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9535.550049] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9542.872417] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9543.019127] Lustre: lustre-OST0001: new disk, initializing [ 9543.023426] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9543.030908] Lustre: 274722:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9544.908842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9544.930349] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9545.088405] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9549.071235] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9557.743208] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9561.155805] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9565.562273] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9565.635050] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9573.345723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9573.354838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9573.363902] Lustre: Skipped 1 previous similar message [ 9573.368852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9577.914475] Lustre: server umount lustre-MDT0000 complete [ 9581.034769] LustreError: 274846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9581.054918] LustreError: 274846:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9581.451902] LustreError: 272400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449707 with bad export cookie 16795476323767692903 [ 9581.465296] LustreError: 272400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9581.466208] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9581.762843] Lustre: server umount lustre-MDT0001 complete [ 9594.458912] Lustre: server umount lustre-OST0000 complete [ 9607.853149] Lustre: server umount lustre-OST0001 complete [ 9622.323972] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 9634.495936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9635.069419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9635.084138] Lustre: Skipped 3 previous similar messages [ 9640.114871] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9649.312760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9653.728960] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9656.541501] Lustre: 278925:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9664.319085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9670.803150] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9674.862178] LustreError: 279281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9674.890479] LustreError: 279281:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9678.939855] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9680.261209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9680.279392] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9684.956468] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9691.208787] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9694.263324] Lustre: 280802:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9698.687278] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9698.774176] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9706.745457] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 11:37:10 (1788449830) [ 9721.311949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9721.316859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9721.319560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9721.319568] Lustre: Skipped 4 previous similar messages [ 9721.358725] Lustre: Skipped 5 previous similar messages [ 9727.404720] Lustre: server umount lustre-MDT0000 complete [ 9730.698072] LustreError: 278927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449856 with bad export cookie 16795476323767699707 [ 9730.700607] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9730.710106] LustreError: 278927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9740.710910] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9740.724564] Lustre: Skipped 12 previous similar messages [ 9745.375177] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9745.705483] Lustre: server umount lustre-MDT0001 complete [ 9756.519807] Lustre: server umount lustre-OST0000 complete [ 9766.809026] Lustre: server umount lustre-OST0001 complete [ 9773.813404] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 9783.968226] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 9831.946625] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 9842.604489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9842.875898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9842.901419] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9842.984243] Lustre: lustre-MDT0000: new disk, initializing [ 9843.058815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9847.314576] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9857.250215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9857.399650] Lustre: 285042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9857.705759] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9862.192887] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9875.185898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9875.439847] Lustre: 286675:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9876.864666] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9883.179911] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9895.026855] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9895.400331] Lustre: 287546:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9895.599588] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9895.611605] Lustre: Skipped 6 previous similar messages [ 9896.874198] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9896.885515] Lustre: Skipped 1 previous similar message [ 9897.032382] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9902.081851] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9910.715531] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9914.948948] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9914.962036] Lustre: Skipped 1 previous similar message [ 9931.756273] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 11:40:54 (1788450054) [ 9934.304274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9934.311888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9934.322471] Lustre: Skipped 3 previous similar messages [ 9934.329252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9936.147712] Lustre: server umount lustre-MDT0000 complete [ 9937.894675] LustreError: 285056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9937.915390] LustreError: 285056:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 9939.025230] LustreError: 286674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450064 with bad export cookie 16795476323767707127 [ 9939.027030] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9939.033079] LustreError: 286674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9939.307925] Lustre: server umount lustre-MDT0001 complete [ 9953.374750] Lustre: server umount lustre-OST0000 complete [ 9967.921056] Lustre: server umount lustre-OST0001 complete [ 9976.182166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9981.551985] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9989.924406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9994.687144] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10001.363267] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10006.751552] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10013.432991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10019.060880] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10026.212567] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10043.276465] loop1: detected capacity change from 0 to 204800000 [10043.541583] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10043.549224] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10043.555826] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10048.549611] loop1: detected capacity change from 0 to 204800000 [10053.369725] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10053.448715] LustreError: 289778:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs3685: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10053.467503] LustreError: 289778:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs3685 [10053.473090] LustreError: 289778:0:(mgs_handler.c:517:mgs_target_reg()) fs3685-MDT0001: Error checking target index: rc = -13 [10053.484640] LustreError: fs3685-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10053.499767] LustreError: 293508:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10053.510236] LustreError: 293508:0:(tgt_mount.c:2129:server_put_super()) no obd fs3685-MDT0001 [10053.515582] LustreError: 293508:0:(tgt_mount.c:116:server_deregister_mount()) fs3685-MDT0001 not registered [10053.553710] Lustre: server umount fs3685-MDT0001 complete [10053.559336] LustreError: 293508:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10060.662185] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10060.746783] Lustre: Setting parameter fs3685-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs3685-MDT0001 [10060.788077] Lustre: srv-fs3685-MDT0001: No data found on store. Initialize space. [10060.790979] Lustre: Skipped 4 previous similar messages [10060.849506] Lustre: fs3685-MDT0001: new disk, initializing [10060.853694] Lustre: Skipped 3 previous similar messages [10060.982424] Lustre: srv-fs3685-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10060.992564] Lustre: 293974:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3685-MDT0001-mdtlov: can't create new llog: rc = -115 [10061.005618] LustreError: 293974:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs3685-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10064.332950] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10070.113832] Lustre: server umount fs3685-MDT0001 complete [10083.467171] loop1: detected capacity change from 0 to 204800000 [10089.391817] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10089.735581] Lustre: 295540:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3685-MDT0001-mdtlov: can't create new llog: rc = -115 [10089.745335] LustreError: 295540:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs3685-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10094.124116] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10098.680885] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10100.335658] Lustre: server umount fs3685-MDT0001 complete [10112.996516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10113.004346] Lustre: Skipped 3 previous similar messages [10113.007548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10124.255134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10124.527508] Lustre: server umount lustre-MDT0000 complete [10133.751808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10133.908150] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10138.488497] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10139.657464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10146.530197] loop1: detected capacity change from 0 to 204800000 [10152.382248] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10152.715436] Lustre: 297991:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3685-MDT0001-mdtlov: can't create new llog: rc = -115 [10152.728643] LustreError: 297991:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs3685-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10157.716346] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10164.625482] Lustre: server umount fs3685-MDT0001 complete [10188.772091] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10188.783676] LustreError: Skipped 2 previous similar messages [10194.216849] Lustre: server umount lustre-OST0000 complete [10213.343421] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10213.639583] Lustre: server umount lustre-MDT0000 complete [10215.397195] LustreError: 289786:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10215.408866] LustreError: 289786:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 106 previous similar messages [10217.122475] LustreError: 289773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450342 with bad export cookie 16795476323767709808 [10217.125481] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10217.130348] LustreError: 289773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10217.530142] Lustre: server umount lustre-MDT0001 complete [10226.978356] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [10229.425293] Key type lgssc unregistered [10229.631826] LNet: 300339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10229.644728] LNetError: 300339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10229.666457] LNet: Removed LNI 192.168.206.114@tcp [10230.305211] Key type .llcrypt unregistered [10230.308602] Key type ._llcrypt unregistered [10248.894445] Key type ._llcrypt registered [10248.897245] Key type .llcrypt registered [10248.989827] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [10250.078290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10250.301628] alg: No test for adler32 (adler32-zlib) [10251.296187] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10251.497460] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [10253.169462] Key type lgssc registered [10254.073730] Lustre: Echo OBD driver; http://www.lustre.org/ [10260.906710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10260.926906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10262.328784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10266.289265] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10271.720977] LustreError: 301615:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10271.743048] LustreError: 301615:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10274.366810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10274.703421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10278.402580] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10285.513767] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10291.171709] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10297.969568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10298.348484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10299.379060] LustreError: 303443:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10304.488648] LustreError: 303446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10304.901608] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10306.541984] LustreError: 303443:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10306.560028] LustreError: 303443:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10311.652849] LustreError: 303524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10311.693352] LustreError: 303524:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10313.117340] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10320.832799] LustreError: 303446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10320.850371] LustreError: 303446:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10330.330330] loop1: detected capacity change from 0 to 204800000 [10330.541605] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10330.556197] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10330.572121] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10336.005802] loop1: detected capacity change from 0 to 204800000 [10337.253667] LustreError: 303524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10337.278073] LustreError: 303524:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [10341.200627] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10341.349892] Lustre: fs32572-OST0000: new disk, initializing [10341.354142] Lustre: srv-fs32572-OST0000: No data found on store. Initialize space. [10341.362526] Lustre: 305348:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10341.441968] Lustre: fs32572-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10349.645020] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10358.595747] Lustre: server umount fs32572-OST0000 complete [10370.334051] loop1: detected capacity change from 0 to 204800000 [10372.036693] LustreError: 304067:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10372.065950] LustreError: 304067:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [10376.301620] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10376.533600] Lustre: fs32572-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10382.281342] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10383.341143] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10388.467141] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10390.123483] loop2: detected capacity change from 0 to 204800000 [10390.377578] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10390.383833] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10390.391746] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10395.641977] loop2: detected capacity change from 0 to 204800000 [10401.110892] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10401.187657] LustreError: fs32572-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10401.200502] LustreError: fs32572-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10401.206665] LustreError: 308386:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10401.211755] LustreError: 308386:0:(tgt_mount.c:2129:server_put_super()) no obd fs32572-OST0001 [10401.217770] LustreError: 308386:0:(tgt_mount.c:116:server_deregister_mount()) fs32572-OST0001 not registered [10401.243508] Lustre: server umount fs32572-OST0001 complete [10401.245746] LustreError: 308386:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10412.127803] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10412.274263] Lustre: fs32572-OST0001: new disk, initializing [10412.278268] Lustre: srv-fs32572-OST0001: No data found on store. Initialize space. [10412.282924] Lustre: 308938:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10412.344641] Lustre: fs32572-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10418.801397] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10429.518470] Lustre: server umount fs32572-OST0000 complete [10438.593349] LustreError: 303524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10438.617154] LustreError: 303524:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [10440.277809] Lustre: server umount fs32572-OST0001 complete [10453.285486] loop1: detected capacity change from 0 to 204800000 [10458.527509] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10458.798269] Lustre: fs32572-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10465.167802] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10470.991509] loop2: detected capacity change from 0 to 204800000 [10476.230218] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10482.168114] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10490.288758] Lustre: server umount fs32572-OST0000 complete [10499.871662] Lustre: server umount fs32572-OST0001 complete [10514.401144] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10514.414773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10514.432165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10516.455881] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10516.468437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10516.469906] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10516.494582] Lustre: Skipped 1 previous similar message [10521.571555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10521.577064] Lustre: Skipped 1 previous similar message [10526.175106] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10526.364116] Lustre: server umount lustre-OST0000 complete [10530.786403] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10530.795482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10530.827734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10536.325607] Lustre: server umount lustre-MDT0000 complete [10540.601616] LustreError: 301595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450666 with bad export cookie 3680858122766192347 [10540.624238] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10540.625993] LustreError: 301595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10541.084869] Lustre: server umount lustre-MDT0001 complete [10553.075926] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [10555.681328] Key type lgssc unregistered [10555.933206] LNet: 314867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10555.941370] LNetError: 314867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10555.958936] LNet: Removed LNI 192.168.206.114@tcp [10556.808392] Key type .llcrypt unregistered [10556.814113] Key type ._llcrypt unregistered [10592.629731] Key type ._llcrypt registered [10592.634122] Key type .llcrypt registered [10592.782583] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [10609.845523] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [10611.399789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10611.445933] alg: No test for adler32 (adler32-zlib) [10612.629490] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10613.116883] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [10614.968566] Key type lgssc registered [10616.428401] Lustre: Echo OBD driver; http://www.lustre.org/ [10652.873710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10652.901593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10654.243549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10654.284846] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10654.375784] Lustre: lustre-MDT0000: new disk, initializing [10654.452599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10654.487253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10660.319052] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10671.183802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10671.343185] Lustre: 318730:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10671.420187] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10671.423275] Lustre: Skipped 1 previous similar message [10671.518697] Lustre: lustre-MDT0001: new disk, initializing [10671.665995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10671.721142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10671.725664] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10676.701572] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10686.648976] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10693.100690] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10700.521926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10700.863339] Lustre: lustre-OST0000: new disk, initializing [10700.868323] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10700.881907] Lustre: 320700:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10701.003629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10707.788966] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10710.634316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10710.649801] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10710.842560] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10717.218270] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10719.269161] Lustre: Mounted lustre-client - version 2.17.56_51_g2aa7cea [10720.304398] Lustre: Unmounted lustre-client [10731.499731] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10731.503173] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10731.524316] LustreError: Skipped 1 previous similar message [10731.525500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10731.558471] Lustre: Skipped 1 previous similar message [10734.562963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10734.580163] Lustre: Skipped 2 previous similar messages [10736.779864] Lustre: server umount lustre-OST0000 complete [10743.267956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10743.275080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10743.287973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10744.802546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10746.749491] Lustre: server umount lustre-MDT0000 complete [10750.530397] LustreError: 319525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450876 with bad export cookie 7007185559316159483 [10750.539016] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10750.545196] LustreError: 319525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10750.971683] Lustre: server umount lustre-MDT0001 complete [10761.766721] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [10764.482564] Key type lgssc unregistered [10769.952789] LNet: 322862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10769.975551] LNetError: 322862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10851.957911] LNet: Removed LNI 192.168.206.114@tcp [10853.224104] Key type .llcrypt unregistered [10853.232281] Key type ._llcrypt unregistered [10874.725864] Key type ._llcrypt registered [10874.737611] Key type .llcrypt registered [10874.855758] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [10892.974734] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [10894.050729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10894.074657] alg: No test for adler32 (adler32-zlib) [10895.321909] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10895.676149] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [10897.383364] Key type lgssc registered [10898.504102] Lustre: Echo OBD driver; http://www.lustre.org/ [10930.157583] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 11:57:33 (1788451053) [10937.952640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10937.985180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10939.319892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10939.344065] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10939.427523] Lustre: lustre-MDT0000: new disk, initializing [10939.531504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10939.560403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10945.862617] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10958.079843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10958.233925] Lustre: 326087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10958.354976] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10958.361924] Lustre: Skipped 1 previous similar message [10958.458786] Lustre: lustre-MDT0001: new disk, initializing [10958.622445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10958.665299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10958.681790] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10963.100527] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10972.906304] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10978.987210] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10987.544119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10987.965311] Lustre: lustre-OST0000: new disk, initializing [10987.976764] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10987.987928] Lustre: 328057:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10988.144621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10994.347889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10994.361352] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10994.550884] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10996.137798] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [11007.059791] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11025.378771] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11025.379494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11025.405353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11025.411254] Lustre: Skipped 1 previous similar message [11030.474965] Lustre: server umount lustre-OST0000 complete [11035.616499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11035.631362] Lustre: Skipped 1 previous similar message [11035.661854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11040.737454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11040.751958] Lustre: Skipped 2 previous similar messages [11041.526382] Lustre: server umount lustre-MDT0000 complete [11045.341349] LustreError: 326081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451171 with bad export cookie 13277928325369604125 [11045.360269] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11045.363310] LustreError: 326081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11045.741691] Lustre: server umount lustre-MDT0001 complete [11058.846235] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [11062.299116] Key type lgssc unregistered [11062.649945] LNet: 329970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11062.665422] LNetError: 329970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11062.696867] LNet: Removed LNI 192.168.206.114@tcp [11063.967920] Key type .llcrypt unregistered [11063.972864] Key type ._llcrypt unregistered [11086.877172] Key type ._llcrypt registered [11086.882312] Key type .llcrypt registered [11087.051867] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [11088.670825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11088.736311] alg: No test for adler32 (adler32-zlib) [11089.962984] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11090.269688] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [11092.031267] Key type lgssc registered [11093.298985] Lustre: Echo OBD driver; http://www.lustre.org/ [11101.591915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11101.637855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11103.301362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11108.117755] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [11112.432072] LustreError: 331242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11112.462375] LustreError: 331242:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11116.646731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11117.040015] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11121.859304] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [11129.365349] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11135.153538] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11143.205058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11143.643585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11150.357759] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [11158.099927] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11171.555464] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11172.959374] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11177.443966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11177.458817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11177.471444] Lustre: Skipped 2 previous similar messages [11177.485316] Lustre: Skipped 2 previous similar messages [11181.545204] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.14@tcp (stopping) [11182.560218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11182.578734] Lustre: Skipped 2 previous similar messages [11182.949923] Lustre: server umount lustre-MDT0000 complete [11186.614639] LustreError: 332841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11187.037864] LustreError: 331224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451312 with bad export cookie 12275662619433587712 [11187.043685] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11187.048603] LustreError: 331224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11187.681085] LustreError: 332841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11187.682965] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11187.720725] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11191.751272] LustreError: 331239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11191.802614] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.14@tcp (stopping) [11193.607177] Lustre: server umount lustre-MDT0001 complete [11197.991993] Lustre: server umount lustre-OST0000 complete [11205.578486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11209.956402] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [11212.412947] Lustre: 335434:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.114@tcp [11214.421943] Lustre: server umount MGS complete [11222.901858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11223.167149] LustreError: 336005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11223.183965] LustreError: 336005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11223.260393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11228.661069] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [11231.671507] LustreError: 336000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11231.711938] LustreError: 336000:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11238.057987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11238.348898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11243.607304] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [11251.742847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11252.203965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11259.069555] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [11267.534563] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11269.228303] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11275.645774] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11277.460380] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11282.913465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11282.922995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11282.948800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11282.967791] Lustre: Skipped 1 previous similar message [11282.977881] Lustre: Skipped 1 previous similar message [11284.687691] Lustre: server umount lustre-OST0000 complete [11289.575851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11289.583483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11294.885890] Lustre: server umount lustre-MDT0000 complete [11298.201405] LustreError: 335986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451423 with bad export cookie 12275662619433588552 [11298.218655] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11298.479282] Lustre: server umount lustre-MDT0001 complete [11309.309655] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [11312.055734] Key type lgssc unregistered [11312.408647] LNet: 339442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11312.415756] LNetError: 339442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11312.431860] LNet: Removed LNI 192.168.206.114@tcp [11313.321979] Key type .llcrypt unregistered [11313.327258] Key type ._llcrypt unregistered [11330.370192] Key type ._llcrypt registered [11330.371787] Key type .llcrypt registered [11330.453765] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [11346.860407] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [11348.574445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11348.625454] alg: No test for adler32 (adler32-zlib) [11349.733454] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11350.002493] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [11351.663183] Key type lgssc registered [11353.509721] Lustre: Echo OBD driver; http://www.lustre.org/ [11391.622140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11391.652960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11392.925354] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11392.965628] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11393.084560] Lustre: lustre-MDT0000: new disk, initializing [11393.144380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11393.153914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11397.539085] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [11408.205407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11408.326326] Lustre: 342604:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11408.376855] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11408.379827] Lustre: Skipped 1 previous similar message [11408.451614] Lustre: lustre-MDT0001: new disk, initializing [11408.571548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11408.638935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11408.645696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11413.369797] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [11422.115647] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11427.283894] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11433.381182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11433.630416] Lustre: lustre-OST0000: new disk, initializing [11433.633816] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11433.639144] Lustre: 344571:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11433.710318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11439.043579] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [11442.272854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11442.283871] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11442.457114] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11448.433978] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11453.145733] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11453.289470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11457.352958] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11457.514470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11463.147042] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11463.149506] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11463.157358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11463.158960] LustreError: Skipped 1 previous similar message [11463.186777] Lustre: Skipped 1 previous similar message [11468.265992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11468.278176] Lustre: Skipped 1 previous similar message [11473.377244] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11473.377884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11473.405074] Lustre: Skipped 1 previous similar message [11473.550692] Lustre: server umount lustre-OST0000 complete [11478.499654] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11478.516170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11478.520468] Lustre: Skipped 2 previous similar messages [11483.618820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11483.635750] Lustre: Skipped 1 previous similar message [11491.295128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11491.589651] Lustre: server umount lustre-MDT0000 complete [11494.738990] LustreError: 342595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451620 with bad export cookie 12415745165740727970 [11494.746533] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11494.754945] LustreError: 342595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11495.195487] Lustre: server umount lustre-MDT0001 complete [11504.101331] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 12:07:07 (1788451627) [11519.690446] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [11528.421939] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [11563.241400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11563.555481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11563.580259] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11563.654908] Lustre: lustre-MDT0000: new disk, initializing [11563.706654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11563.720710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11568.763550] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [11579.486966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11579.639118] Lustre: 349335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11579.702674] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11579.705561] Lustre: Skipped 1 previous similar message [11579.787372] Lustre: lustre-MDT0001: new disk, initializing [11579.900762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11579.936774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11579.955201] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11584.339299] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [11593.137737] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11598.515401] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11604.290552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11604.590973] Lustre: lustre-OST0000: new disk, initializing [11604.595707] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11604.600987] Lustre: 351302:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11604.688027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11610.246233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11610.253533] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11610.384838] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11610.484444] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [11620.019593] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11651.552375] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11651.565243] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11651.567017] LustreError: Skipped 1 previous similar message [11651.568326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11651.568332] Lustre: Skipped 2 previous similar messages [11651.589989] Lustre: Skipped 2 previous similar messages [11656.528100] Lustre: server umount lustre-OST0000 complete [11661.798329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11661.804217] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11661.814073] Lustre: Skipped 1 previous similar message [11661.820841] LustreError: Skipped 1 previous similar message [11666.914436] LustreError: 350618:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11666.935036] LustreError: 350618:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11667.082218] Lustre: server umount lustre-MDT0000 complete [11670.944478] LustreError: 349325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451796 with bad export cookie 12415745165740728859 [11670.946758] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11670.963417] LustreError: 349325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11671.311747] Lustre: server umount lustre-MDT0001 complete [11680.952378] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [11683.418340] Key type lgssc unregistered [11683.685185] LNet: 353476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11683.694971] LNetError: 353476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11684.712026] LNet: Removed LNI 192.168.206.114@tcp [11685.598631] Key type .llcrypt unregistered [11685.600188] Key type ._llcrypt unregistered [11704.468501] Key type ._llcrypt registered [11704.470916] Key type .llcrypt registered [11704.558558] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [11705.211922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11705.229495] alg: No test for adler32 (adler32-zlib) [11706.454079] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11706.733306] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [11708.464272] Key type lgssc registered [11709.459597] Lustre: Echo OBD driver; http://www.lustre.org/ [11716.629332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11716.663487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11718.257778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11722.772255] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [11727.845536] LustreError: 354750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11727.879333] LustreError: 354750:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11730.470189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11731.025658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11736.734511] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [11744.976409] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11750.757303] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11758.197329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11758.656618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11765.317401] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [11773.730416] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11791.845300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11791.851083] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11791.853478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11791.865777] Lustre: Skipped 1 previous similar message [11794.642458] Lustre: server umount lustre-OST0000 complete [11802.080506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11802.106800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11802.116842] Lustre: Skipped 2 previous similar messages [11807.201886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11807.209384] Lustre: Skipped 1 previous similar message [11812.325321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11812.338056] Lustre: Skipped 1 previous similar message [11813.347915] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11813.668720] Lustre: server umount lustre-MDT0000 complete [11817.443663] LustreError: 355478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11817.467985] LustreError: 355478:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11817.598257] LustreError: 354732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451943 with bad export cookie 11731485423998530389 [11817.605522] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11817.607276] LustreError: 354732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11818.232868] Lustre: server umount lustre-MDT0001 complete [11828.745738] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [11831.492234] Key type lgssc unregistered [11831.884259] LNet: 360714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11831.895493] LNetError: 360714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11831.929906] LNet: Removed LNI 192.168.206.114@tcp [11833.028144] Key type .llcrypt unregistered [11833.031167] Key type ._llcrypt unregistered [11851.380293] Key type ._llcrypt registered [11851.383902] Key type .llcrypt registered [11851.501698] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [11852.569792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11852.648032] alg: No test for adler32 (adler32-zlib) [11853.689422] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11853.984072] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [11855.719137] Key type lgssc registered [11857.311831] Lustre: Echo OBD driver; http://www.lustre.org/ [11865.156716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11865.183621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11866.601065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11871.825976] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [11876.839267] LustreError: 364590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11876.866292] LustreError: 364590:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11879.667167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11880.064981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11884.502565] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [11890.915901] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11896.149869] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11902.971024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11903.347867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11909.877174] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [11916.794884] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11924.962880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11924.969886] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11924.980836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11927.010418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11927.013857] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11927.032872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11927.047929] Lustre: Skipped 1 previous similar message [11930.738462] Lustre: server umount lustre-OST0000 complete [11936.225960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11936.235368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11936.247406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11940.821410] Lustre: server umount lustre-MDT0000 complete [11942.374379] LustreError: 364589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11942.405103] LustreError: 364589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11944.265587] LustreError: 364505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452069 with bad export cookie 15925059429567280019 [11944.274207] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11944.275431] LustreError: 364505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11944.728308] Lustre: server umount lustre-MDT0001 complete [11954.286475] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [11956.872890] Key type lgssc unregistered [11957.193430] LNet: 372233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11957.207424] LNetError: 372233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11957.230428] LNet: Removed LNI 192.168.206.114@tcp [11958.170387] Key type .llcrypt unregistered [11958.174490] Key type ._llcrypt unregistered [11973.689954] Key type ._llcrypt registered [11973.692321] Key type .llcrypt registered [11973.744795] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [11974.446913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11974.455786] alg: No test for adler32 (adler32-zlib) [11975.670763] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11975.893784] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [11977.505133] Key type lgssc registered [11978.852432] Lustre: Echo OBD driver; http://www.lustre.org/ [11987.150069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11987.179246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11988.845256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11994.173899] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [11999.203463] LustreError: 376063:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11999.227804] LustreError: 376063:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12002.394102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12002.726917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12007.126836] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [12015.068628] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12020.979198] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12027.468806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12028.042684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12036.687584] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [12044.966705] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12055.013332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12055.015247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12055.020395] LustreError: Skipped 1 previous similar message [12055.020988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12055.026981] Lustre: Skipped 1 previous similar message [12055.055526] Lustre: Skipped 1 previous similar message [12058.779902] Lustre: server umount lustre-OST0000 complete [12064.228732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12064.239575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12064.257553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12065.254596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12065.262614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12065.267719] Lustre: Skipped 1 previous similar message [12068.412223] Lustre: server umount lustre-MDT0000 complete [12070.368344] LustreError: 376065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12070.393764] LustreError: 376065:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12071.649245] LustreError: 375985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452197 with bad export cookie 12347936202732768209 [12071.661931] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12072.101730] Lustre: server umount lustre-MDT0001 complete [12081.859153] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [12084.211598] Key type lgssc unregistered [12084.540526] LNet: 383614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12084.544872] LNetError: 383614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12084.570628] LNet: Removed LNI 192.168.206.114@tcp [12085.521429] Key type .llcrypt unregistered [12085.525827] Key type ._llcrypt unregistered [12101.836403] Key type ._llcrypt registered [12101.838591] Key type .llcrypt registered [12101.990285] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [12102.769373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12102.841284] alg: No test for adler32 (adler32-zlib) [12103.867710] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12104.116584] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [12105.823082] Key type lgssc registered [12106.839507] Lustre: Echo OBD driver; http://www.lustre.org/ [12113.460356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12113.478640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12114.956318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12119.167641] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [12125.153104] LustreError: 387511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12125.164487] LustreError: 387511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12127.818066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12128.291250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12134.207944] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [12141.752807] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12147.738309] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12156.168727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12156.546171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12161.937380] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [12168.032411] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12180.453896] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12180.461120] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12180.479595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12183.527487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12183.546476] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12183.556648] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12186.527907] Lustre: server umount lustre-OST0000 complete [12193.762737] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12193.773640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12193.779706] Lustre: Skipped 1 previous similar message [12193.788438] Lustre: Skipped 2 previous similar messages [12196.077232] Lustre: server umount lustre-MDT0000 complete [12198.799501] LustreError: 387425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452324 with bad export cookie 15902628806198038108 [12198.813550] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12199.063675] Lustre: server umount lustre-MDT0001 complete [12208.650462] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [12211.384046] Key type lgssc unregistered [12211.626748] LNet: 393471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12211.636359] LNetError: 393471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12211.675641] LNet: Removed LNI 192.168.206.114@tcp [12212.554697] Key type .llcrypt unregistered [12212.557384] Key type ._llcrypt unregistered [12290.207375] Key type ._llcrypt registered [12290.211581] Key type .llcrypt registered [12290.316403] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [12290.577267] Key type .llcrypt unregistered [12290.578943] Key type ._llcrypt unregistered [12343.343587] Key type ._llcrypt registered [12343.353057] Key type .llcrypt registered [12343.461627] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [12343.696255] Key type .llcrypt unregistered [12343.699058] Key type ._llcrypt unregistered [12367.039230] Key type ._llcrypt registered [12367.042729] Key type .llcrypt registered [12367.150087] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [12367.412812] Key type .llcrypt unregistered [12367.415788] Key type ._llcrypt unregistered [12387.953657] Key type ._llcrypt registered [12387.956114] Key type .llcrypt registered [12388.066217] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [12388.384592] Key type .llcrypt unregistered [12388.387204] Key type ._llcrypt unregistered [12454.990514] Key type ._llcrypt registered [12454.994274] Key type .llcrypt registered [12455.087325] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [12455.328867] Key type .llcrypt unregistered [12455.335435] Key type ._llcrypt unregistered [12478.825188] Key type ._llcrypt registered [12478.827455] Key type .llcrypt registered [12478.922651] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [12479.176271] Key type .llcrypt unregistered [12479.180880] Key type ._llcrypt unregistered [12501.192766] Key type ._llcrypt registered [12501.194767] Key type .llcrypt registered [12501.285409] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [12501.520336] Key type .llcrypt unregistered [12501.522565] Key type ._llcrypt unregistered [12524.427166] Key type ._llcrypt registered [12524.428966] Key type .llcrypt registered [12524.504510] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [12524.827301] Key type .llcrypt unregistered [12524.830309] Key type ._llcrypt unregistered [12547.396077] Key type ._llcrypt registered [12547.398755] Key type .llcrypt registered [12547.486592] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [12547.749310] Key type .llcrypt unregistered [12547.757524] Key type ._llcrypt unregistered [12573.847305] Key type ._llcrypt registered [12573.850543] Key type .llcrypt registered [12573.957380] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [12574.485294] Key type .llcrypt unregistered [12574.487048] Key type ._llcrypt unregistered [12633.524486] Key type ._llcrypt registered [12633.528184] Key type .llcrypt registered [12633.619093] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [12633.925433] Key type .llcrypt unregistered [12633.933447] Key type ._llcrypt unregistered [12660.139359] Key type ._llcrypt registered [12660.145455] Key type .llcrypt registered [12660.257384] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [12660.625329] Key type .llcrypt unregistered [12660.630731] Key type ._llcrypt unregistered [12686.092983] Key type ._llcrypt registered [12686.104099] Key type .llcrypt registered [12686.280776] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [12686.704392] Key type .llcrypt unregistered [12686.707888] Key type ._llcrypt unregistered [12739.762166] Key type ._llcrypt registered [12739.764635] Key type .llcrypt registered [12739.863097] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [12754.786966] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [12755.572922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12755.639053] alg: No test for adler32 (adler32-zlib) [12756.714113] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12757.000354] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [12758.706522] Key type lgssc registered [12759.845688] Lustre: Echo OBD driver; http://www.lustre.org/ [12788.718449] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12788.731975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12790.009464] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12790.052668] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12790.114236] Lustre: tst250fs-MDT0000: new disk, initializing [12790.183987] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12790.199555] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12795.645297] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [12805.036228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12805.129935] Lustre: 413857:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12805.177857] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12805.181427] Lustre: Skipped 1 previous similar message [12805.263770] Lustre: tst250fs-MDT0001: new disk, initializing [12805.346322] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12805.370167] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12805.379387] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12808.813619] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [12813.365476] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12819.552638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12819.817496] Lustre: tst250fs-OST0000: new disk, initializing [12819.821713] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12819.830374] Lustre: 415491:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12819.932614] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12823.881048] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12823.892611] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12824.000424] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12824.561500] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [12834.185125] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12834.331464] Lustre: tst250fs-OST0001: new disk, initializing [12834.334762] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12834.340645] Lustre: 416360:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12834.415229] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12840.085903] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12840.102854] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12840.236943] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12840.367070] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [12845.515880] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12847.984407] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [12847.988143] Lustre: Skipped 2 previous similar messages [12866.025410] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12866.031182] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12866.050388] Lustre: Skipped 3 previous similar messages [12866.062133] Lustre: Skipped 2 previous similar messages [12870.796258] Lustre: server umount tst250fs-MDT0000 complete [12875.092198] LustreError: 413848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453000 with bad export cookie 10209314606804725529 [12875.096990] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12875.106729] LustreError: 413848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12875.777454] Lustre: server umount tst250fs-MDT0001 complete [12890.525680] Lustre: server umount tst250fs-OST0000 complete [12891.436676] Lustre: 412080:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788453001/real 1788453001] req@ffff95470b339180 x1875328769263744/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788453017 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12891.476833] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12894.269808] Lustre: server umount tst250fs-OST0001 complete [12910.447826] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [12918.591117] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [12950.600686] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:31:13 (1788453073) [12952.020947] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12966.833405] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [12975.760940] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [13004.037525] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12694 sec ========= 12:32:07 (1788453127) [13005.670393] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:32:08 (1788453128) === [13008.508920] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:32:11 (1788453131) === [13043.295569] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [13046.166735] Key type lgssc unregistered [13046.460885] LNet: 425500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13046.494154] LNetError: 425500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13046.519048] LNet: Removed LNI 192.168.206.114@tcp [13047.758262] Key type .llcrypt unregistered [13047.760071] Key type ._llcrypt unregistered