[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1239700995 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002010] Switched APIC routing to physical x2apic. [ 0.003014] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006035] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007011] pid_max: default: 32768 minimum: 301 [ 0.008156] LSM: Security Framework initializing [ 0.009068] Yama: becoming mindful. [ 0.010031] SELinux: Initializing. [ 0.011094] *** VALIDATE selinux *** [ 0.022071] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028096] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029153] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031118] *** VALIDATE tmpfs *** [ 0.032480] *** VALIDATE proc *** [ 0.033280] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.036219] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038110] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040029] Spectre V2 : User space: Vulnerable [ 0.041012] Speculative Store Bypass: Vulnerable [ 0.044117] debug: unmapping init [mem 0xffffffff97659000-0xffffffff97660fff] [ 0.046366] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047659] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048021] ... version: 2 [ 0.049010] ... bit width: 48 [ 0.050016] ... generic registers: 4 [ 0.051011] ... value mask: 0000ffffffffffff [ 0.052017] ... max period: 00007fffffffffff [ 0.053012] ... fixed-purpose events: 3 [ 0.054011] ... event mask: 000000070000000f [ 0.055277] rcu: Hierarchical SRCU implementation. [ 0.057471] smp: Bringing up secondary CPUs ... [ 0.058615] x86: Booting SMP configuration: [ 0.059020] .... node #0, CPUs: #1 #2 #3 [ 0.072019] smp: Brought up 1 node, 4 CPUs [ 0.074033] smpboot: Max logical packages: 1 [ 0.075015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.186165] node 0 deferred pages initialised in 108ms [ 0.195184] devtmpfs: initialized [ 0.197244] x86/mm: Memory block size: 128MB [ 0.208094] gcov: version magic: 0x41383552 [ 0.211518] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.213435] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.217560] pinctrl core: initialized pinctrl subsystem [ 0.220280] [ 0.221008] ************************************************************* [ 0.224016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.228134] ** ** [ 0.231012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.234013] ** ** [ 0.237015] ** This means that this kernel is built to expose internal ** [ 0.241016] ** IOMMU data structures, which may compromise security on ** [ 0.244012] ** your system. ** [ 0.247012] ** ** [ 0.250012] ** If you see this message and you are not debugging the ** [ 0.253012] ** kernel, report this immediately to your vendor! ** [ 0.256159] ** ** [ 0.260013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.262013] ************************************************************* [ 0.266210] NET: Registered protocol family 16 [ 0.268651] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.273057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.277075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.283150] cpuidle: using governor menu [ 0.285000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.290213] PCI: Using configuration type 1 for base access [ 0.294238] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.306316] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.308019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.313328] cryptd: max_cpu_qlen set to 1000 [ 0.316227] ACPI: Added _OSI(Module Device) [ 0.319017] ACPI: Added _OSI(Processor Device) [ 0.322015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.325016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.333696] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.348176] ACPI: Interpreter enabled [ 0.351065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.362016] ACPI: Using IOAPIC for interrupt routing [ 0.369775] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.376000] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.399582] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.405409] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.410020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.416077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.438062] acpiphp: Slot [2] registered [ 0.446062] acpiphp: Slot [5] registered [ 0.450090] acpiphp: Slot [6] registered [ 0.457042] acpiphp: Slot [7] registered [ 0.463167] acpiphp: Slot [8] registered [ 0.469177] acpiphp: Slot [9] registered [ 0.477022] acpiphp: Slot [10] registered [ 0.479983] acpiphp: Slot [3] registered [ 0.488512] acpiphp: Slot [4] registered [ 0.493147] acpiphp: Slot [11] registered [ 0.494000] acpiphp: Slot [12] registered [ 0.500132] acpiphp: Slot [13] registered [ 0.505127] acpiphp: Slot [14] registered [ 0.513025] acpiphp: Slot [15] registered [ 0.521132] acpiphp: Slot [16] registered [ 0.526283] acpiphp: Slot [17] registered [ 0.531139] acpiphp: Slot [18] registered [ 0.535113] acpiphp: Slot [19] registered [ 0.548145] acpiphp: Slot [20] registered [ 0.555123] acpiphp: Slot [21] registered [ 0.562130] acpiphp: Slot [22] registered [ 0.568146] acpiphp: Slot [23] registered [ 0.572051] acpiphp: Slot [24] registered [ 0.581140] acpiphp: Slot [25] registered [ 0.585127] acpiphp: Slot [26] registered [ 0.592231] acpiphp: Slot [27] registered [ 0.598126] acpiphp: Slot [28] registered [ 0.606269] acpiphp: Slot [29] registered [ 0.617207] acpiphp: Slot [30] registered [ 0.621098] acpiphp: Slot [31] registered [ 0.629068] PCI host bridge to bus 0000:00 [ 0.637025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.646028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.655024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.660023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.669975] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.676025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.680150] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.688838] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.696000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.926115] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.937067] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.940016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.945017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.949018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.953964] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.958316] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.961429] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.965827] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.970012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.984020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.991015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.997329] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 1.158025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 1.169026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 1.183022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 1.195084] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 1.202034] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 1.209022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 1.229030] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 1.240591] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 1.247024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 1.253021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 1.266086] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 1.273328] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 1.278023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.283018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.299027] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 1.309000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.317026] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.327025] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.343024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.356287] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.362023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.369021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.385024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.397593] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.400412] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.403393] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.405453] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.408230] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.412167] iommu: Default domain type: Passthrough [ 1.414584] SCSI subsystem initialized [ 1.416162] ACPI: bus type USB registered [ 1.417126] usbcore: registered new interface driver usbfs [ 1.419105] usbcore: registered new interface driver hub [ 1.421095] usbcore: registered new device driver usb [ 1.423178] pps_core: LinuxPPS API ver. 1 registered [ 1.425012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.427066] PTP clock support registered [ 1.428209] EDAC MC: Ver: 3.0.0 [ 1.430387] PCI: Using ACPI for IRQ routing [ 1.432062] NetLabel: Initializing [ 1.433026] NetLabel: domain hash size = 128 [ 1.435013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.437100] NetLabel: unlabeled traffic allowed by default [ 1.439052] vgaarb: loaded [ 1.441027] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.442010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.448625] clocksource: Switched to clocksource kvm-clock [ 1.607652] VFS: Disk quotas dquot_6.6.0 [ 1.610169] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.613862] *** VALIDATE ramfs *** [ 1.614796] *** VALIDATE hugetlbfs *** [ 1.616405] pnp: PnP ACPI init [ 1.618991] pnp: PnP ACPI: found 6 devices [ 1.635344] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.638882] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.641177] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.643371] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.645942] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.648435] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.651447] NET: Registered protocol family 2 [ 1.654305] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.659581] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.662640] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.670577] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.680135] TCP: Hash tables configured (established 65536 bind 65536) [ 1.688101] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.695660] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.702390] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.710181] NET: Registered protocol family 1 [ 1.757712] RPC: Registered named UNIX socket transport module. [ 1.761119] RPC: Registered udp transport module. [ 1.764585] RPC: Registered tcp transport module. [ 1.768081] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.801127] NET: Registered protocol family 44 [ 1.802997] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.804921] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.807023] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.809804] PCI: CLS 0 bytes, default 64 [ 1.811784] Unpacking initramfs... [ 2.369676] hrtimer: interrupt took 10045820 ns [ 5.066278] debug: unmapping init [mem 0xffff8a23fcc54000-0xffff8a23fffbffff] [ 5.072731] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 5.077314] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 5.083621] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.832830] Initialise system trusted keyrings [ 5.834049] Key type blacklist registered [ 5.836522] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.845517] zbud: loaded [ 5.847865] *** VALIDATE nfs *** [ 5.848729] *** VALIDATE nfs4 *** [ 5.849983] pstore: using deflate compression [ 5.852844] Platform Keyring initialized [ 5.953860] NET: Registered protocol family 38 [ 5.956552] Key type asymmetric registered [ 5.957905] Asymmetric key parser 'x509' registered [ 5.959561] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.962106] io scheduler mq-deadline registered [ 5.963464] io scheduler kyber registered [ 5.966525] io scheduler bfq registered [ 5.968332] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.972887] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.046164] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.351342] ACPI: Power Button [PWRF] [ 6.549879] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.558181] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.707663] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.739439] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.757796] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.836351] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.769904] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.888765] Non-volatile memory driver v1.3 [ 7.891156] Linux agpgart interface v0.103 [ 8.000523] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 8.008194] vda: detected capacity change from 0 to 74702848 [ 8.077197] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 8.085502] vdb: detected capacity change from 0 to 1073741824 [ 8.121146] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.126044] vdc: detected capacity change from 0 to 2621440000 [ 8.155965] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.160247] vdd: detected capacity change from 0 to 2621440000 [ 8.742059] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.745945] vde: detected capacity change from 0 to 4294967296 [ 8.927920] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.931424] vdf: detected capacity change from 0 to 4294967296 [ 8.940337] libphy: Fixed MDIO Bus: probed [ 8.947861] usbcore: registered new interface driver usbserial_generic [ 8.950675] usbserial: USB Serial support registered for generic [ 8.953670] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 8.959269] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 8.961522] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 8.964807] mousedev: PS/2 mouse device common for all mice [ 8.973780] rtc_cmos 00:05: RTC can wake from S4 [ 8.976084] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 8.982123] rtc_cmos 00:05: registered as rtc0 [ 8.982200] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 8.982257] intel_pstate: CPU model not supported [ 8.986722] hid: raw HID events driver (C) Jiri Kosina [ 8.993163] usbcore: registered new interface driver usbhid [ 8.995504] usbhid: USB HID core driver [ 8.997666] drop_monitor: Initializing network drop monitor service [ 9.001347] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 9.005428] Initializing XFRM netlink socket [ 9.007826] NET: Registered protocol family 10 [ 9.012275] Segment Routing with IPv6 [ 9.013974] NET: Registered protocol family 17 [ 9.016506] mpls_gso: MPLS GSO support [ 9.018297] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 9.025800] RAS: Correctable Errors collector initialized. [ 9.028064] AVX version of gcm_enc/dec engaged. [ 9.029852] AES CTR mode by8 optimization enabled [ 10.514368] sched_clock: Marking stable (10514285209, 0)->(12840481261, -2326196052) [ 10.610913] registered taskstats version 1 [ 10.613895] Loading compiled-in X.509 certificates [ 10.617746] zswap: loaded using pool lzo/zbud [ 10.648982] Key type big_key registered [ 10.666793] Key type encrypted registered [ 10.668642] ima: No TPM chip found, activating TPM-bypass! [ 10.671211] ima: Allocated hash algorithm: sha1 [ 10.673829] ima: No architecture policies found [ 10.677291] evm: Initialising EVM extended attributes: [ 10.679555] evm: security.selinux [ 10.680740] evm: security.ima [ 10.682169] evm: security.capability [ 10.683818] evm: HMAC attrs: 0x1 [ 10.686962] rtc_cmos 00:05: setting system clock to 2026-08-13 22:15:30 UTC (1786659330) [ 10.694868] debug: unmapping init [mem 0xffffffff98603000-0xffffffff987fffff] [ 10.699353] debug: unmapping init [mem 0xffffffff97382000-0xffffffff97658fff] [ 10.709154] Write protecting the kernel read-only data: 28672k [ 10.713678] debug: unmapping init [mem 0xffffffff95a03000-0xffffffff95bfffff] [ 10.716798] debug: unmapping init [mem 0xffffffff96314000-0xffffffff963fffff] [ 10.808304] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.825167] systemd[1]: Detected virtualization kvm. [ 10.828521] systemd[1]: Detected architecture x86-64. [ 10.832311] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 10.878700] systemd[1]: No hostname configured. [ 10.881346] systemd[1]: Set hostname to . [ 10.885097] random: systemd: uninitialized urandom read (16 bytes read) [ 10.888482] systemd[1]: Initializing machine ID from random generator. [ 11.621684] random: systemd: uninitialized urandom read (16 bytes read) [ 11.626425] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 11.997779] random: systemd: uninitialized urandom read (16 bytes read) [ 12.001180] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 12.081131] random: systemd: uninitialized urandom read (16 bytes read) [ 12.083698] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 12.107172] random: systemd: uninitialized urandom read (16 bytes read) [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ 13.102167] urandom_read: 4 callbacks suppressed [ 13.102185] random: systemd: uninitialized urandom read (16 bytes read) Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 16.717675] device-mapper: uevent: version 1.0.3 [ 16.729857] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 19.579578] random: fast init done [ 19.699551] virtio_net virtio0 ens2: renamed from eth0 [ 19.950751] scsi host0: ata_piix [ 20.049425] scsi host1: ata_piix [ 20.056868] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 20.068928] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 25.200947] random: crng init done [ 29.047304] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 31.915041] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 36.694821] printk: systemd: 23 output lines suppressed due to ratelimiting [ 37.999722] SELinux: Disabled at runtime. [ 38.275463] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 38.304105] systemd[1]: Detected virtualization kvm. [ 38.314782] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 41.348154] systemd[1]: initrd-switch-root.service: Succeeded. [ 41.358970] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 41.406619] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 41.433411] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 41.452473] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 41.502326] systemd[1]: Starting Journal Service... Starting Journal Service... [ 41.530893] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [ 41.634826] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 44.277307] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 46.263205] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 46.522826] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 47.468503] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 47.654854] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…only root support (10s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (13s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (15s / no limit)[ 56.294863] Key type dns_resolver registered [*** ] A start job is running for Configur…only root support (15s / no limit) [** ] A start job is running for Configur…only root support (16s / no limit) [* ] A start job is running for Configur…only root support (16s / no limit)[ 57.766694] NFS: Registering the id_resolver key type [ 57.778687] Key type id_resolver registered [ 57.785681] Key type id_legacy registered [** ] A start job is running for Configur…only root support (17s / no limit) [*** ] A start job is running for Configur…only root support (17s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg159-server login: [ 127.702035] libcfs: loading out-of-tree module taints kernel. [ 127.766261] Key type ._llcrypt registered [ 127.768397] Key type .llcrypt registered [ 127.869314] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 146.000807] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 147.706727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 147.719753] alg: No test for adler32 (adler32-zlib) [ 149.124286] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 150.147448] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 152.039205] Key type lgssc registered [ 153.726281] Lustre: Echo OBD driver; http://www.lustre.org/ [ 167.793529] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 211.321821] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 227.748547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 227.775779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 229.115624] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 229.155794] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 229.257498] Lustre: lustre-MDT0000: new disk, initializing [ 229.355898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 229.374167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 233.509965] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.818392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 248.032172] Lustre: 6503:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 248.151747] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 248.162609] Lustre: Skipped 1 previous similar message [ 248.316785] Lustre: lustre-MDT0001: new disk, initializing [ 248.438251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 248.481626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 248.495856] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 252.649896] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 258.056158] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 267.503459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 267.904270] Lustre: lustre-OST0000: new disk, initializing [ 267.908338] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 267.917693] Lustre: 8443:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 267.996894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 272.087418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 272.104727] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 272.160288] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 275.944561] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 290.643339] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 290.763562] Lustre: lustre-OST0001: new disk, initializing [ 290.768691] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 290.781793] Lustre: 9514:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 290.905531] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 296.984286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 296.994742] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 297.054932] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 297.596409] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 310.489479] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 318.779940] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 326.221185] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing check_logdir /tmp/testlogs/ [ 331.326696] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing yml_node [ 336.506381] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 339.718463] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 342.776297] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 344.757395] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 13 18:21:03 EDT 2026 [ 365.972399] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 368.499508] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 387.042314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 387.051982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 387.072067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 389.092191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 389.092570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 389.108889] Lustre: Skipped 1 previous similar message [ 389.129213] Lustre: Skipped 1 previous similar message [ 392.835304] Lustre: server umount lustre-MDT0000 complete [ 396.804890] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786659716 with bad export cookie 9719665466997043616 [ 396.806240] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 396.827729] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 399.329248] LustreError: 6512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 399.334738] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 399.341621] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 399.341631] Lustre: Skipped 2 previous similar messages [ 399.346703] LustreError: 6512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 399.365444] Lustre: Skipped 2 previous similar messages [ 401.382722] LustreError: 6513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 401.382873] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 401.425323] LustreError: 6513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 401.448531] Lustre: Skipped 2 previous similar messages [ 403.279405] Lustre: server umount lustre-MDT0001 complete [ 407.952553] Lustre: server umount lustre-OST0000 complete [ 412.436435] Lustre: server umount lustre-OST0001 complete [ 420.201674] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 431.769913] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 472.847700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 473.120800] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 473.146388] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 473.219628] Lustre: lustre-MDT0000: new disk, initializing [ 473.279966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 473.294878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 479.291653] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 497.230106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 497.630825] Lustre: 15073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 497.838693] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 497.853709] Lustre: Skipped 1 previous similar message [ 498.036799] Lustre: lustre-MDT0001: new disk, initializing [ 498.372224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 498.534826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 498.572066] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 504.860625] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 516.230312] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 523.243774] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 533.253505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 533.800917] Lustre: lustre-OST0000: new disk, initializing [ 533.805219] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 533.813922] Lustre: 17044:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 533.996656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 535.192053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 535.210299] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 535.426697] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 543.725681] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 555.549989] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 561.510536] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 561.728717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 567.250626] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 567.446424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 571.362674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 571.387512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 571.391390] Lustre: Skipped 1 previous similar message [ 571.408551] Lustre: Skipped 1 previous similar message [ 576.042732] Lustre: server umount lustre-OST0000 complete [ 580.064595] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 580.073273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 580.093125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 594.399131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 594.720527] Lustre: server umount lustre-MDT0000 complete [ 596.961697] LustreError: 15085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 596.978301] LustreError: 15085:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 598.227558] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786659918 with bad export cookie 9719665466997045156 [ 598.232202] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 598.241851] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 598.607050] Lustre: server umount lustre-MDT0001 complete [ 604.822464] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 18:25:23 (1786659923) [ 624.233548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 624.738235] LustreError: 19924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 624.838623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 629.540154] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 630.250686] LustreError: 19925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 639.362312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 639.460618] LustreError: 19925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 639.489060] LustreError: 19925:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 644.731243] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 653.032644] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 659.126996] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 661.248659] Lustre: Failing over lustre-MDT0000 [ 661.494157] Lustre: server umount lustre-MDT0000 complete [ 665.380172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 665.390114] LustreError: 19919:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 665.390979] Lustre: Skipped 2 previous similar messages [ 665.422492] LustreError: 19919:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 680.105692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 680.189144] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 680.418591] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 684.693238] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 685.475197] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 685.485188] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 685.499031] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 693.955950] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 700.896307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 700.904463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 700.904653] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 700.910306] Lustre: Skipped 1 previous similar message [ 700.918419] Lustre: Skipped 7 previous similar messages [ 704.433663] Lustre: server umount lustre-MDT0000 complete [ 706.017030] LustreError: 20928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 706.029566] LustreError: 20928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 708.617074] LustreError: 19905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786660028 with bad export cookie 9719665466997046570 [ 708.625183] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 708.632844] LustreError: 19905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 709.062936] Lustre: server umount lustre-MDT0001 complete [ 719.914800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 720.618200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 720.624621] Lustre: Skipped 2 previous similar messages [ 726.579232] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 737.070895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 742.276032] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 750.879436] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 757.312703] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 760.611370] Lustre: Failing over lustre-MDT0000 [ 762.721232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 762.735857] Lustre: Skipped 1 previous similar message [ 762.947611] Lustre: server umount lustre-MDT0000 complete [ 773.089150] LustreError: 23265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 773.122449] LustreError: 23265:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 778.207573] Lustre: 3639:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786660082/real 1786660082] req@ffff8a24739a5880 x1873448239566464/t0(0) o400->MGC192.168.201.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786660098 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 778.243833] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 782.554303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 789.485870] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x86e3303ec7fd80d4 [ 789.497703] Lustre: MGC192.168.201.159@tcp: Connection restored to 0@lo (at 0@lo) [ 789.500516] Lustre: Skipped 1 previous similar message [ 789.885977] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 794.966830] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 795.109475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 795.110498] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 795.167030] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 806.204562] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 810.464620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 810.475860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 816.729469] Lustre: server umount lustre-MDT0000 complete [ 820.440673] LustreError: 23246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786660140 with bad export cookie 9719665466997047508 [ 820.443171] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 820.451658] LustreError: 23246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 820.823247] Lustre: server umount lustre-MDT0001 complete [ 830.738401] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 18:29:09 (1786660149) [ 848.872520] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 860.167078] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 900.250813] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 910.550918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 910.821450] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 910.841891] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 910.906824] Lustre: lustre-MDT0000: new disk, initializing [ 910.977161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 910.981942] Lustre: Skipped 2 previous similar messages [ 910.994399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 915.908962] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 926.372545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 926.528795] Lustre: 29641:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 926.818459] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 930.932445] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 941.294386] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 946.133558] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 952.040736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 952.245844] Lustre: 31612:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 954.284695] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 954.339897] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 954.411861] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 954.425818] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 958.280845] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 967.982344] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 978.867545] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1000.417199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1000.418665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1000.422846] LustreError: Skipped 1 previous similar message [ 1000.438977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1000.439396] Lustre: Skipped 3 previous similar messages [ 1000.459738] Lustre: Skipped 3 previous similar messages [ 1002.504123] Lustre: server umount lustre-OST0000 complete [ 1008.613956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1020.383874] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1020.619722] Lustre: server umount lustre-MDT0000 complete [ 1020.897100] LustreError: 29650:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1020.907992] LustreError: 29650:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 1024.549749] LustreError: 29634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786660344 with bad export cookie 9719665466997047921 [ 1024.551712] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1024.569160] LustreError: 29634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1025.071387] Lustre: server umount lustre-MDT0001 complete [ 1039.436437] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 1043.641401] Key type lgssc unregistered [ 1043.945612] LNet: 33596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.956742] LNetError: 33596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1043.977790] LNet: Removed LNI 192.168.201.159@tcp [ 1045.004904] Key type .llcrypt unregistered [ 1045.010109] Key type ._llcrypt unregistered [ 1076.878607] Key type ._llcrypt registered [ 1076.880622] Key type .llcrypt registered [ 1076.956166] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 1078.176181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1078.310247] alg: No test for adler32 (adler32-zlib) [ 1079.485980] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 1079.653329] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 1081.264155] Key type lgssc registered [ 1081.906635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1088.767749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1088.794736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1090.197105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1094.803857] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1099.747972] LustreError: 35010:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1099.760149] LustreError: 35010:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1104.865767] LustreError: 35011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1105.247871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1105.598368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1110.038388] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1118.219229] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1124.256966] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1131.464863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1131.758790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1139.358861] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1147.805481] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1151.230422] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 1165.711231] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 18:34:44 (1786660484) [ 1168.040382] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1170.193202] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1178.075677] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 18:34:56 (1786660496) [ 1188.255735] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 18:35:06 (1786660506) [ 1189.356713] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1199.482861] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1199.490735] Lustre: Skipped 2 previous similar messages [ 1202.109476] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 18:35:20 (1786660520) [ 1204.307432] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1204.315545] Lustre: Skipped 2 previous similar messages [ 1213.451740] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1213.459378] Lustre: Skipped 4 previous similar messages [ 1227.591098] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 18:35:46 (1786660546) [ 1230.573292] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1234.509383] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1243.439331] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 18:36:02 (1786660562) [ 1245.427690] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1245.432422] Lustre: Skipped 6 previous similar messages [ 1260.300414] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 18:36:18 (1786660578) [ 1278.436490] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1278.440370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1278.476474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1279.459497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1279.461058] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1279.483641] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1281.976498] Lustre: server umount lustre-OST0000 complete [ 1289.697743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1289.701207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1289.706847] Lustre: Skipped 1 previous similar message [ 1289.717690] Lustre: Skipped 2 previous similar messages [ 1292.555440] Lustre: server umount lustre-MDT0000 complete [ 1294.818957] LustreError: 35007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1294.842943] LustreError: 35007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1296.738122] LustreError: 34992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786660616 with bad export cookie 4137791723506694736 [ 1296.752052] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1296.761271] LustreError: 34992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1297.175891] Lustre: server umount lustre-MDT0001 complete [ 1309.349429] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 1312.100161] Key type lgssc unregistered [ 1312.397611] LNet: 42867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1312.401546] LNetError: 42867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1312.412812] LNet: Removed LNI 192.168.201.159@tcp [ 1313.308372] Key type .llcrypt unregistered [ 1313.310355] Key type ._llcrypt unregistered [ 1333.160549] Key type ._llcrypt registered [ 1333.164318] Key type .llcrypt registered [ 1333.248302] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 1350.111064] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 1351.190438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1351.364765] alg: No test for adler32 (adler32-zlib) [ 1352.435919] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 1352.646846] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 1354.303483] Key type lgssc registered [ 1355.452373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1389.695339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1389.744100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1390.962164] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1390.993680] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1391.056479] Lustre: lustre-MDT0000: new disk, initializing [ 1391.146572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1391.165243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1395.962918] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1406.051527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1406.206477] Lustre: 46023:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1406.263172] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1406.268968] Lustre: Skipped 1 previous similar message [ 1406.366379] Lustre: lustre-MDT0001: new disk, initializing [ 1406.462324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1406.499920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1406.516499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1411.108953] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1420.400337] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1425.392794] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1431.669244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1431.970826] Lustre: lustre-OST0000: new disk, initializing [ 1431.980563] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1431.987503] Lustre: 47992:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1432.068369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1438.490964] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1441.888475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1441.901263] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1442.004580] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1448.584317] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1454.658921] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1455.001761] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1460.689364] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1460.923893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1467.883889] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1467.885647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1467.894705] LustreError: Skipped 1 previous similar message [ 1467.908929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1467.915803] Lustre: Skipped 1 previous similar message [ 1469.402952] Lustre: server umount lustre-OST0000 complete [ 1478.116952] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1478.126623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1478.139114] Lustre: Skipped 1 previous similar message [ 1478.142207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1479.993327] Lustre: server umount lustre-MDT0000 complete [ 1484.078343] LustreError: 47991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786660803 with bad export cookie 15886508299925502565 [ 1484.084114] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1484.091780] LustreError: 47991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1484.346635] Lustre: server umount lustre-MDT0001 complete [ 1494.594862] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 18:40:13 (1786660813) [ 1501.915454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1502.300739] LustreError: 50452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1502.322586] LustreError: 50452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1502.385608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1506.878871] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1507.815147] LustreError: 50453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1512.931050] LustreError: 50453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1515.896135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1516.182557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1521.082310] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1528.398612] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1534.433669] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1540.886303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1541.100446] LustreError: 52272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1541.226309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1547.592537] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1555.330200] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1560.047749] LustreError: 52981:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1561.105208] Lustre: Modifying parameter general.timeout=1 in log params [ 1562.130507] Lustre: Modifying parameter general.timeout=2 in log params [ 1564.930518] Lustre: Modifying parameter general.timeout=5 in log params [ 1564.934421] Lustre: Skipped 2 previous similar messages [ 1569.515854] Lustre: Modifying parameter general.timeout=9 in log params [ 1569.519721] Lustre: Skipped 3 previous similar messages [ 1578.666449] Lustre: Modifying parameter general.timeout=18 in log params [ 1578.669920] Lustre: Skipped 8 previous similar messages [ 1595.209506] Lustre: Modifying parameter general.timeout=38 in log params [ 1595.212674] Lustre: Skipped 19 previous similar messages [ 1627.528253] Lustre: Modifying parameter general.timeout=81 in log params [ 1627.531448] Lustre: Skipped 42 previous similar messages [ 1673.656740] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 18:43:12 (1786660992) [ 1676.029906] Lustre: 59918:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1677.476993] Lustre: 60016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1680.630982] Lustre: 60229:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1680.641785] Lustre: 60229:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1684.753093] Lustre: 60426:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1684.764684] Lustre: 60426:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1692.244525] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 in log params [ 1692.254875] Lustre: Skipped 61 previous similar messages [ 1714.886589] Lustre: 61799:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1714.915082] LustreError: 61799:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1720.086586] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 18:43:58 (1786661038) [ 1736.160340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1736.167282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1736.177191] Lustre: Skipped 1 previous similar message [ 1736.184066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1736.187304] Lustre: Skipped 1 previous similar message [ 1740.349484] Lustre: server umount lustre-MDT0000 complete [ 1741.797991] LustreError: 50448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1741.818491] LustreError: 50448:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1744.944946] LustreError: 58077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786661064 with bad export cookie 15886508299925503447 [ 1744.946876] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1744.950039] LustreError: 58077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1745.378375] Lustre: server umount lustre-MDT0001 complete [ 1760.610299] Lustre: server umount lustre-OST0000 complete [ 1787.413565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1787.612949] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1787.923892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1793.148510] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1801.398558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1801.532531] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1801.573297] Lustre: 64329:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1806.119494] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1813.469435] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1820.005451] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1827.294980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1827.566835] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1827.887365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1827.894177] Lustre: Skipped 1 previous similar message [ 1834.430978] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1842.645346] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1844.745167] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1844.756848] Lustre: Skipped 10 previous similar messages [ 1844.818197] Lustre: 66896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1853.931844] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1853.944486] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1853.957411] Lustre: Skipped 2 previous similar messages [ 1853.968805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1853.974497] Lustre: Skipped 2 previous similar messages [ 1858.484216] Lustre: server umount lustre-OST0000 complete [ 1863.148906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1863.173336] LustreError: Skipped 1 previous similar message [ 1863.175881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1863.186349] Lustre: Skipped 1 previous similar message [ 1863.193550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1863.200256] Lustre: Skipped 2 previous similar messages [ 1868.659128] Lustre: server umount lustre-MDT0000 complete [ 1869.797331] LustreError: 64342:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1869.816920] LustreError: 64342:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1872.585767] LustreError: 64321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786661192 with bad export cookie 15886508299925506415 [ 1872.598087] LustreError: 64321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1872.598611] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1873.066821] Lustre: server umount lustre-MDT0001 complete [ 1883.758582] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 1886.393602] Key type lgssc unregistered [ 1886.707243] LNet: 68105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1886.712264] LNetError: 68105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1886.725862] LNet: Removed LNI 192.168.201.159@tcp [ 1887.445874] Key type .llcrypt unregistered [ 1887.449697] Key type ._llcrypt unregistered [ 1911.189524] Key type ._llcrypt registered [ 1911.191990] Key type .llcrypt registered [ 1911.277351] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 1912.337138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1912.493662] alg: No test for adler32 (adler32-zlib) [ 1913.771272] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 1914.142555] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 1915.879165] Key type lgssc registered [ 1917.353049] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1926.941647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1926.971986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1928.544291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1933.555985] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1933.795298] LustreError: 69517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1933.807929] LustreError: 69517:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1938.928290] LustreError: 69518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1942.326857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1943.068366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1949.909754] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1959.691675] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1967.360292] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1977.369579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1978.235031] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1986.017726] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1996.121723] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2008.338892] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2019.296853] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2019.305179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2019.317130] Lustre: Skipped 2 previous similar messages [ 2019.346226] Lustre: Skipped 2 previous similar messages [ 2024.417871] LustreError: 69517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2024.429206] LustreError: 69517:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2024.959512] Lustre: server umount lustre-MDT0000 complete [ 2029.217933] LustreError: 69500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786661349 with bad export cookie 15512615353017948610 [ 2029.249816] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2029.540808] LustreError: 71952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2029.548965] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2029.761630] Lustre: server umount lustre-MDT0001 complete [ 2036.910609] Lustre: server umount lustre-OST0000 complete [ 2065.218632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2065.510595] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2065.527753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2065.812088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2070.506190] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2079.887102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2079.958662] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2079.982901] Lustre: 74392:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2080.448779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2084.901202] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2093.649166] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2100.718697] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2107.756292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2107.970907] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2108.260754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2115.352975] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2124.564715] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2127.016149] Lustre: 76969:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2128.977397] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2133.987655] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2133.994089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2134.007664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2138.606658] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2138.616349] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2138.643923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2140.188728] Lustre: server umount lustre-OST0000 complete [ 2144.230671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2144.234058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2144.238902] Lustre: Skipped 1 previous similar message [ 2144.246263] Lustre: Skipped 2 previous similar messages [ 2149.345067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2149.350712] Lustre: Skipped 2 previous similar messages [ 2151.872768] Lustre: server umount lustre-MDT0000 complete [ 2154.465258] LustreError: 74400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2154.487458] LustreError: 74400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2155.087405] LustreError: 76971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786661474 with bad export cookie 15512615353017949457 [ 2155.099919] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2155.104378] LustreError: 76971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2155.458778] Lustre: server umount lustre-MDT0001 complete [ 2164.381093] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 2166.926411] Key type lgssc unregistered [ 2167.167346] LNet: 78180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2167.172789] LNetError: 78180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2167.216304] LNet: Removed LNI 192.168.201.159@tcp [ 2168.561284] Key type .llcrypt unregistered [ 2168.564309] Key type ._llcrypt unregistered [ 2199.847161] Key type ._llcrypt registered [ 2199.850403] Key type .llcrypt registered [ 2200.014516] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 2201.364742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2201.461260] alg: No test for adler32 (adler32-zlib) [ 2202.801416] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 2203.147527] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 2204.823377] Key type lgssc registered [ 2207.381789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2219.338775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2219.375459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2221.119761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2227.716955] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2230.243186] LustreError: 79598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2230.283553] LustreError: 79598:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2235.382072] LustreError: 79599:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2240.284317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2240.495147] LustreError: 79598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2241.082058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2246.439483] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2254.382519] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2260.738243] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2269.380598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2270.503246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2278.498588] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2289.375511] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2300.433965] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2301.628202] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2303.473105] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2303.483149] Lustre: Skipped 1 previous similar message [ 2306.108160] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2306.114858] Lustre: Skipped 2 previous similar messages [ 2310.552745] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2310.555601] Lustre: Skipped 3 previous similar messages [ 2319.319988] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2319.330441] Lustre: Skipped 8 previous similar messages [ 2335.879234] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2335.884822] Lustre: Skipped 14 previous similar messages [ 2368.091642] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2368.094782] Lustre: Skipped 29 previous similar messages [ 2413.233593] Lustre: 87713:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2416.814087] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 18:55:34 (1786661734) [ 2418.964381] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2422.190320] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 18:55:39 (1786661739) [ 2429.408686] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2429.412228] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2429.442883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2430.446723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2430.447795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2430.492989] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2434.869794] Lustre: server umount lustre-OST0000 complete [ 2440.698678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2440.722550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2440.733336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2440.733346] Lustre: Skipped 1 previous similar message [ 2445.856872] LustreError: 82101:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2445.986507] Lustre: server umount lustre-MDT0000 complete [ 2450.913993] LustreError: 82101:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2450.954982] LustreError: 82101:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2451.144112] LustreError: 79580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786661770 with bad export cookie 8107749998107213685 [ 2451.159985] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2451.168877] LustreError: 79580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2451.749813] Lustre: server umount lustre-MDT0001 complete [ 2469.896318] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 2473.640719] Key type lgssc unregistered [ 2473.937270] LNet: 88987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2473.947482] LNetError: 88987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2473.977353] LNet: Removed LNI 192.168.201.159@tcp [ 2475.547660] Key type .llcrypt unregistered [ 2475.552758] Key type ._llcrypt unregistered [ 2480.145521] Key type ._llcrypt registered [ 2480.147411] Key type .llcrypt registered [ 2480.248178] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2485.921677] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules [ 2487.126919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2487.341157] alg: No test for adler32 (adler32-zlib) [ 2488.466959] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 2488.791634] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 2490.544048] Key type lgssc registered [ 2492.133914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2493.874410] LustreError: 90205:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2512.842698] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 2516.038192] LustreError: 90205:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2525.450343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2525.468362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2527.216649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2533.119543] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2536.419262] LustreError: 91207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2536.447809] LustreError: 91207:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2541.543764] LustreError: 91208:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2543.627492] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 18:57:42 (1786661862) [ 2546.673931] LustreError: 91207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2550.343421] LustreError: 91240:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2550.364896] LustreError: 91240:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2550.392994] LustreError: 91240:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 2551.099945] Lustre: server umount lustre-MDT0000 complete [ 2566.892939] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 2569.510577] Key type lgssc unregistered [ 2569.789477] LNet: 92737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2569.796301] LNetError: 92737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2569.811766] LNet: Removed LNI 192.168.201.159@tcp [ 2570.848893] Key type .llcrypt unregistered [ 2570.852435] Key type ._llcrypt unregistered [ 2591.484310] Key type ._llcrypt registered [ 2591.487837] Key type .llcrypt registered [ 2591.595879] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 2592.656620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2592.846712] alg: No test for adler32 (adler32-zlib) [ 2593.912238] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 2594.193305] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 2595.967201] Key type lgssc registered [ 2597.500430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2606.535872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2606.552444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2608.329743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2613.675199] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2617.825231] LustreError: 93955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2617.837554] LustreError: 93955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2622.189970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2623.021796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2627.655491] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2637.117913] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2643.349940] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2651.524495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2652.325315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2660.040907] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2673.035973] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2735.116588] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 19:00:53 (1786662053) [ 2737.501574] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2739.428899] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 19:00:58 (1786662058) [ 2744.806405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2744.809563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2744.822413] Lustre: Skipped 1 previous similar message [ 2744.843747] Lustre: Skipped 2 previous similar messages [ 2749.934645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2749.957362] Lustre: Skipped 2 previous similar messages [ 2751.348560] Lustre: server umount lustre-MDT0000 complete [ 2755.041117] LustreError: 93955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2755.070027] LustreError: 93955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2755.435683] LustreError: 93936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786662075 with bad export cookie 12934660146977127572 [ 2755.437702] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2755.467285] LustreError: 93936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2755.741173] Lustre: server umount lustre-MDT0001 complete [ 2771.089031] Lustre: server umount lustre-OST0000 complete [ 2790.266569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2791.373474] LustreError: 98271:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2791.932877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2797.033261] LustreError: 98272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2797.961197] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2802.155128] LustreError: 98271:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2808.478236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2809.281716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2815.810315] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2825.240979] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2832.977810] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2849.557255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2849.893644] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2849.911704] LustreError: 98261:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2849.941154] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2849.971382] LustreError: 100354:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2849.989210] LustreError: 100354:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2849.995142] LustreError: 100354:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2850.194568] Lustre: server umount lustre-OST0000 complete [ 2850.203654] LustreError: 100354:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2861.173817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2861.360832] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2861.369656] LustreError: 98260:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2861.419276] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2861.429397] LustreError: 100822:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2861.435840] LustreError: 100822:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2861.446691] LustreError: 100822:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2861.497405] Lustre: server umount lustre-OST0000 complete [ 2861.502689] LustreError: 100822:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2871.412696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2871.662685] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2871.671073] Lustre: Found index 0 for lustre-OST0000, updating log [ 2871.677504] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2871.891782] Lustre: lustre-OST0000: new disk, initializing [ 2871.901899] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2871.912634] Lustre: 101357:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2872.166173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2882.028504] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2887.086897] Lustre: Failing over lustre-OST0000 [ 2887.329610] Lustre: server umount lustre-OST0000 complete [ 2895.841919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2895.862419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2895.882546] Lustre: Skipped 1 previous similar message [ 2895.911205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2896.878134] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2901.985895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2902.003376] Lustre: Skipped 3 previous similar messages [ 2903.137737] Lustre: server umount lustre-MDT0000 complete [ 2904.992738] Lustre: 93266:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786662208/real 1786662208] req@000000003de6167c x1873450802198016/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786662224 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2905.045655] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2905.068379] LustreError: 98272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2905.121677] LustreError: 98272:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2908.127342] Lustre: 93266:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786662211/real 1786662211] req@000000007eab2156 x1873450802198912/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786662227 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2909.841120] LustreError: 98252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786662229 with bad export cookie 12934660146977130316 [ 2909.851857] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2909.866522] LustreError: 98252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2910.190425] Lustre: server umount lustre-MDT0001 complete [ 2920.899581] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 19:03:59 (1786662239) [ 2943.681353] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 2956.950828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2957.364228] LustreError: 103726:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2957.496509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2962.160172] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2973.709907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2979.943702] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2982.980421] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2990.256919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2991.272251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2991.280345] Lustre: Skipped 1 previous similar message [ 2991.295868] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2998.463649] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2999.435203] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3002.645193] Lustre: lustre-OST0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3002.648039] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3002.669977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 3010.479771] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3010.641740] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3010.886655] Lustre: lustre-OST0001: new disk, initializing [ 3010.893965] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3010.917882] Lustre: 106048:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3018.448873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3018.468322] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3018.648619] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3020.828175] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3030.449463] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3039.136329] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3045.188037] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.59@tcp (stopping) [ 3046.369606] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3046.377427] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3051.335160] Lustre: server umount lustre-MDT0001 complete [ 3054.573106] LustreError: 103722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3054.597337] LustreError: 103722:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3062.927073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3063.101132] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3063.107610] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3063.219607] Lustre: 103713:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3064.142308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3064.145512] Lustre: Skipped 1 previous similar message [ 3064.890888] LustreError: 107596:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3064.906615] LustreError: 107596:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3064.919158] LustreError: 103718:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3065.131087] LustreError: 107615:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.159@tcp: cfg command failed: rc = -17 [ 3065.147549] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3065.147549] [ 3069.441219] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3069.453776] Lustre: Skipped 1 previous similar message [ 3069.712629] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3081.685146] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 19:06:39 (1786662399) [ 3103.225646] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 3121.973087] Lustre: 109664:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3144.125297] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3148.534379] Lustre: 110799:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3248.612637] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3248.613309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3248.633758] Lustre: Skipped 4 previous similar messages [ 3248.644939] Lustre: Skipped 6 previous similar messages [ 3253.220447] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3255.083581] Lustre: server umount lustre-MDT0000 complete [ 3258.857033] LustreError: 103726:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3258.914588] LustreError: 103726:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 3259.930647] LustreError: 103706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786662579 with bad export cookie 12934660146977131324 [ 3259.948876] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3260.815640] Lustre: server umount lustre-MDT0001 complete [ 3276.291154] Lustre: server umount lustre-OST0000 complete [ 3279.327454] Lustre: 93265:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786662583/real 1786662583] req@ffff8a246ce7a680 x1873450802472704/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786662599 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3279.374300] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3279.392240] Lustre: Skipped 1 previous similar message [ 3280.834354] Lustre: server umount lustre-OST0001 complete [ 3286.443222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3295.264668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3302.780625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3315.092870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3341.072329] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 3353.748023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3353.777411] Lustre: lustre-MDT0000: reset Object Index mappings [ 3354.901049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3363.421129] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3376.181535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3376.205380] Lustre: lustre-MDT0001: reset Object Index mappings [ 3382.989861] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3386.758972] Lustre: 115899:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3393.724601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3399.458060] LustreError: 116254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3399.467528] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3399.482775] LustreError: 116254:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3400.051439] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3401.534186] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:232 to 0x280000400:257) [ 3408.358573] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3414.532527] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3414.546470] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 3416.919534] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3425.447528] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3429.498333] Lustre: 117776:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3491.400588] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 19:13:30 (1786662810) [ 3519.971371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3519.978923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3519.996077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3519.999916] Lustre: Skipped 5 previous similar messages [ 3524.869752] Lustre: server umount lustre-MDT0000 complete [ 3529.219887] LustreError: 115501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786662849 with bad export cookie 12934660146977178399 [ 3529.245464] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3529.249853] LustreError: 115501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3536.047751] Lustre: server umount lustre-MDT0001 complete [ 3548.106749] Lustre: server umount lustre-OST0000 complete [ 3558.590148] Lustre: server umount lustre-OST0001 complete [ 3566.044550] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 3584.158065] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 3626.040869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3626.207182] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3626.227318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3626.285484] Lustre: lustre-MDT0000: new disk, initializing [ 3626.346687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3626.355099] Lustre: Skipped 3 previous similar messages [ 3626.368993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3630.839562] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3637.949381] Lustre: server umount lustre-MDT0000 complete [ 3647.822411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3648.178309] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3648.188651] Lustre: 122071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3648.198261] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3653.807618] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3659.183649] Lustre: server umount lustre-MDT0000 complete [ 3667.999174] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 19:16:26 (1786662986) [ 3669.676931] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3671.587683] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 19:16:30 (1786662990) [ 3690.431982] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 3700.391700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3705.434886] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3713.530158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3713.653928] Lustre: 124137:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3713.707924] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3713.711551] Lustre: Skipped 1 previous similar message [ 3713.838820] Lustre: lustre-MDT0001: new disk, initializing [ 3714.002473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3714.027975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3718.838775] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3723.386581] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3730.478907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3730.745718] Lustre: lustre-OST0000: new disk, initializing [ 3730.750370] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3730.755157] Lustre: 125715:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3732.747423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3732.762871] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3732.959334] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3737.550621] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3751.856966] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3752.068559] Lustre: lustre-OST0001: new disk, initializing [ 3752.080601] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3752.104575] Lustre: 126587:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3753.730702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3753.752018] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3753.970760] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3760.893732] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3772.499523] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3777.181565] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4225.287996] Lustre: 133012:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4225.303876] Lustre: 133012:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4225.326631] Lustre: 133012:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4225.343315] Lustre: 133012:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4225.361869] Lustre: 133012:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4225.376446] Lustre: 133012:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4524.947465] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:30:42 (1786663842) [ 4540.389683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4540.399132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4540.402317] Lustre: Skipped 8 previous similar messages [ 4540.412693] Lustre: Skipped 11 previous similar messages [ 4543.008636] Lustre: server umount lustre-MDT0000 complete [ 4545.506884] LustreError: 127384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4545.555188] LustreError: 127384:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 4547.709354] LustreError: 124129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786663867 with bad export cookie 12934660146977249316 [ 4547.713399] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4547.720724] LustreError: 124129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4550.626350] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4550.627814] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4550.641821] Lustre: Skipped 1 previous similar message [ 4550.651557] Lustre: Skipped 1 previous similar message [ 4562.401327] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4562.957872] Lustre: server umount lustre-MDT0001 complete [ 4582.371639] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4582.549302] Lustre: server umount lustre-OST0000 complete [ 4601.311166] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4601.598886] Lustre: server umount lustre-OST0001 complete [ 4609.942268] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 4621.547550] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 4668.165653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4668.449618] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4668.488326] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4668.622347] Lustre: lustre-MDT0000: new disk, initializing [ 4668.756606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4668.770119] Lustre: Skipped 5 previous similar messages [ 4668.813556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4674.326600] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 4686.947100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4687.103064] Lustre: 136075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4687.143390] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4687.146130] Lustre: Skipped 1 previous similar message [ 4687.201261] Lustre: lustre-MDT0001: new disk, initializing [ 4687.301658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4687.312052] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4693.154797] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 4704.938512] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4710.803829] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4717.572203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4717.772048] Lustre: lustre-OST0000: new disk, initializing [ 4717.775862] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4717.781312] Lustre: 138044:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4719.119759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4719.139428] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4719.255692] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4724.289432] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 4734.913368] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4746.087428] Lustre: lustre-MDD0000: changelog on [ 4748.835267] Lustre: lustre-MDD0001: changelog on [ 6714.514205] Lustre: lustre-MDD0001: changelog off [ 6716.438244] Lustre: 139500:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6717.613995] Lustre: lustre-MDD0000: changelog off [ 6726.128083] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6726.139091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6726.157141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6726.160975] Lustre: Skipped 4 previous similar messages [ 6729.699381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6729.708953] Lustre: Skipped 2 previous similar messages [ 6731.144757] Lustre: server umount lustre-OST0000 complete [ 6734.821486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6734.835959] Lustre: Skipped 1 previous similar message [ 6734.840762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6739.936989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6739.949815] Lustre: Skipped 2 previous similar messages [ 6741.129456] Lustre: server umount lustre-MDT0000 complete [ 6744.550463] LustreError: 136066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786666064 with bad export cookie 12934660146978027884 [ 6744.554791] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6744.563231] LustreError: 136066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6744.922600] Lustre: server umount lustre-MDT0001 complete [ 6757.385893] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 6760.156623] Key type lgssc unregistered [ 6760.439201] LNet: 147211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6760.448651] LNetError: 147211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6761.510492] LNet: Removed LNI 192.168.201.159@tcp [ 6762.443185] Key type .llcrypt unregistered [ 6762.446426] Key type ._llcrypt unregistered [ 6783.428635] Key type ._llcrypt registered [ 6783.431508] Key type .llcrypt registered [ 6783.566852] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 6799.946985] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 6801.573061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6801.592446] alg: No test for adler32 (adler32-zlib) [ 6802.723536] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 6802.984206] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 6804.647723] Key type lgssc registered [ 6805.861881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6841.560946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6841.588306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6843.005534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6843.092523] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6843.262755] Lustre: lustre-MDT0000: new disk, initializing [ 6843.393937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6843.424982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6850.039966] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6863.500552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6863.609887] Lustre: 150418:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6863.678236] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6863.682469] Lustre: Skipped 1 previous similar message [ 6863.735913] Lustre: lustre-MDT0001: new disk, initializing [ 6863.810851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6863.846546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6863.859221] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6870.099741] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6881.458689] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6887.990133] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6895.933300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6896.187184] Lustre: lustre-OST0000: new disk, initializing [ 6896.196365] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6896.208183] Lustre: 152384:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6896.347759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6901.841933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6901.863679] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6902.086689] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6904.368038] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6913.911897] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6921.272124] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_module obdecho/obdecho [ 6923.630428] LustreError: 153387:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6923.638653] LustreError: 153387:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6926.286205] LustreError: 153484:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6940.640959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6940.646045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6940.661704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6943.202483] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6943.204610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6946.550818] Lustre: server umount lustre-MDT0000 complete [ 6950.026477] LustreError: 150407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786666269 with bad export cookie 8090236556163921312 [ 6950.039092] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6950.476055] Lustre: server umount lustre-MDT0001 complete [ 6964.854544] Lustre: server umount lustre-OST0000 complete [ 6987.879574] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 6999.910409] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 7040.278937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7040.590168] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7040.642643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7040.732560] Lustre: lustre-MDT0000: new disk, initializing [ 7040.871289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7040.907493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7045.665369] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7056.862643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7057.055940] Lustre: 157205:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7057.136360] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7057.143528] Lustre: Skipped 1 previous similar message [ 7057.338469] Lustre: lustre-MDT0001: new disk, initializing [ 7057.546560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7057.607077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7057.627086] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7062.992365] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7073.256942] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7079.584827] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7086.518722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7086.774324] Lustre: lustre-OST0000: new disk, initializing [ 7086.778740] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7086.786450] Lustre: 159173:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7086.880201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7093.508179] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7096.385638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7096.396969] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7096.523628] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7103.308138] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7109.438033] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7109.799239] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7115.965518] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7116.238992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7122.405834] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7122.414117] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7122.423445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7122.431317] Lustre: Skipped 2 previous similar messages [ 7122.452362] Lustre: Skipped 3 previous similar messages [ 7124.811453] Lustre: server umount lustre-OST0000 complete [ 7129.056113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7129.065355] LustreError: Skipped 1 previous similar message [ 7129.070435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7129.086847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7137.762171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7137.776349] Lustre: Skipped 3 previous similar messages [ 7143.391195] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7143.690135] Lustre: server umount lustre-MDT0000 complete [ 7147.776745] LustreError: 157196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786666467 with bad export cookie 8090236556163922271 [ 7147.790350] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7147.795839] LustreError: 157196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7148.222595] Lustre: server umount lustre-MDT0001 complete [ 7152.540318] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 20:14:30 (1786666470) [ 7154.340233] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7156.939918] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 20:14:35 (1786666475) [ 7166.961356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7167.696562] LustreError: 161586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7167.729784] LustreError: 161586:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7167.862931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7173.096767] LustreError: 161587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7173.562048] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7177.256298] LustreError: 161586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7182.306155] LustreError: 161587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7182.982438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7188.834131] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7197.777060] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7206.362634] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7214.237149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7214.852954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7214.866295] Lustre: Skipped 1 previous similar message [ 7222.141457] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7231.296391] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7252.775435] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.59@tcp (stopping) [ 7252.789237] Lustre: Skipped 2 previous similar messages [ 7252.967063] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7252.991706] Lustre: Skipped 2 previous similar messages [ 7254.551461] Lustre: server umount lustre-MDT0000 complete [ 7257.854403] LustreError: 164078:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7258.379432] LustreError: 161567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786666578 with bad export cookie 8090236556163923146 [ 7258.403544] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7258.406785] LustreError: 161567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7263.205736] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7263.215301] Lustre: Skipped 1 previous similar message [ 7265.179400] Lustre: server umount lustre-MDT0001 complete [ 7272.925089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7287.474579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7288.047238] LustreError: 165454:0:(llog.c:1655:llog_backup()) MGC192.168.201.159@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7288.057507] Lustre: 165454:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.159@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7292.801380] LustreError: 165459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7292.841541] LustreError: 165459:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7309.281851] LustreError: 165461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7309.287632] LustreError: 165461:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 7344.376037] LustreError: 165460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7344.392237] LustreError: 165460:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 25 previous similar messages [ 7348.703814] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7348.714908] Lustre: 165454:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.159@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7389.704700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7394.200759] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7394.815053] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7402.820748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7405.026161] LustreError: 166195:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 7405.040806] LustreError: 166195:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000e54c806b x1873455215676160/t0(0) o250->MGC192.168.201.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1786666724 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7405.064508] LustreError: 166195:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 7406.051939] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x70464b48af07d607 [ 7406.060723] Lustre: MGC192.168.201.159@tcp: Connection restored to 0@lo (at 0@lo) [ 7411.704432] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7411.796880] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7420.205751] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7422.377688] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7429.372101] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7431.335129] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7452.652394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7452.654369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7452.658532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7452.658541] Lustre: Skipped 5 previous similar messages [ 7452.709242] Lustre: Skipped 1 previous similar message [ 7453.644322] Lustre: server umount lustre-MDT0000 complete [ 7457.762244] LustreError: 165459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7457.772450] LustreError: 165459:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 47 previous similar messages [ 7458.547834] LustreError: 161566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786666778 with bad export cookie 8090236556163929607 [ 7458.556712] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7458.559376] LustreError: 161566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7458.868844] Lustre: server umount lustre-MDT0001 complete [ 7475.124208] Lustre: server umount lustre-OST0000 complete [ 7502.604316] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 7519.288598] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 7561.749691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7562.017541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7562.049136] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7562.146723] Lustre: lustre-MDT0000: new disk, initializing [ 7562.270565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7562.281938] Lustre: Skipped 1 previous similar message [ 7562.324186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7567.662104] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7580.682746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7580.795693] Lustre: 170581:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7581.036866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7587.064214] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7600.482459] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7610.061530] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7620.010445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7620.432557] Lustre: lustre-OST0000: new disk, initializing [ 7620.434776] Lustre: Skipped 1 previous similar message [ 7620.449489] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7620.457329] Lustre: Skipped 2 previous similar messages [ 7620.468644] Lustre: 172549:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7629.368022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7629.381115] Lustre: Skipped 1 previous similar message [ 7629.394311] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7629.490663] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7629.533095] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7641.441813] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7647.294918] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7647.605658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7654.075324] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7654.387769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7660.514389] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7660.516282] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7660.521280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7660.521290] Lustre: Skipped 2 previous similar messages [ 7660.532555] LustreError: Skipped 1 previous similar message [ 7660.577619] Lustre: Skipped 2 previous similar messages [ 7671.263207] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7671.682513] Lustre: server umount lustre-OST0000 complete [ 7678.437486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7682.557648] Lustre: server umount lustre-MDT0000 complete [ 7686.105678] LustreError: 170572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786667005 with bad export cookie 8090236556163946435 [ 7686.111738] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7686.117034] LustreError: 170572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7686.516536] Lustre: server umount lustre-MDT0001 complete [ 7696.568687] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 20:23:35 (1786667015) [ 7704.092103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7704.820865] LustreError: 174965:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7709.200326] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7718.932815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7724.236913] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7732.373205] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7738.944154] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7747.986786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7756.249313] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7764.986253] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7778.518053] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 20:24:56 (1786667096) [ 7785.441660] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7785.447518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7785.462435] Lustre: Skipped 2 previous similar messages [ 7785.471206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7785.498483] Lustre: Skipped 9 previous similar messages [ 7790.121454] Lustre: server umount lustre-OST0000 complete [ 7800.658477] Lustre: server umount lustre-MDT0000 complete [ 7804.692045] LustreError: 174945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786667124 with bad export cookie 8090236556163947317 [ 7804.695862] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7804.703105] LustreError: 174945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7805.047301] Lustre: server umount lustre-MDT0001 complete [ 7817.200510] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 7820.096473] Key type lgssc unregistered [ 7820.349149] LNet: 178700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7820.353769] LNetError: 178700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7820.367667] LNet: Removed LNI 192.168.201.159@tcp [ 7821.154070] Key type .llcrypt unregistered [ 7821.156430] Key type ._llcrypt unregistered [ 7841.050347] Key type ._llcrypt registered [ 7841.052616] Key type .llcrypt registered [ 7841.183896] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 7842.608330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7842.798448] alg: No test for adler32 (adler32-zlib) [ 7843.873519] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 7844.132755] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 7845.791124] Key type lgssc registered [ 7847.067205] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7853.553237] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7853.576361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7869.087913] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7874.207657] LustreError: 179945:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.159@tcp: failed processing log, type 4: rc = -110 [ 7879.391448] LustreError: 179945:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7879.404762] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7879.408811] LustreError: MGC192.168.201.159@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7879.416236] LustreError: 179945:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7879.422553] LustreError: 179945:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7879.427976] LustreError: 179945:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7879.505697] Lustre: server umount lustre-OST0000 complete [ 7879.513026] LustreError: 179945:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7889.657750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7891.221359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7896.256955] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7900.647918] LustreError: 180448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7900.656311] LustreError: 180448:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7905.767154] LustreError: 180447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7907.075212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7907.775989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7914.976403] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7926.247136] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7933.544114] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7941.357622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7941.486551] LustreError: 182264:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7941.496565] Lustre: 182264:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.159@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7941.674660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7948.642973] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7956.965353] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7963.105620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7963.109276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7963.129639] Lustre: Skipped 1 previous similar message [ 7963.150135] Lustre: Skipped 2 previous similar messages [ 7965.573441] Lustre: server umount lustre-MDT0000 complete [ 7968.230478] LustreError: 181187:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7968.270782] LustreError: 181187:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7969.621446] LustreError: 180429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786667289 with bad export cookie 12644752089516485713 [ 7969.634341] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7970.089291] Lustre: server umount lustre-MDT0001 complete [ 7985.480389] Lustre: server umount lustre-OST0000 complete [ 7997.480766] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 8009.315104] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 8054.900563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8055.424127] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8055.473131] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8055.610640] Lustre: lustre-MDT0000: new disk, initializing [ 8055.706265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8055.752556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8061.218350] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8075.275183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8075.575137] Lustre: 185794:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8075.659847] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8075.663900] Lustre: Skipped 1 previous similar message [ 8075.788743] Lustre: lustre-MDT0001: new disk, initializing [ 8076.061565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8076.123254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8076.136548] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8081.442566] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8091.743626] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8098.380379] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8105.553494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8105.965860] Lustre: lustre-OST0000: new disk, initializing [ 8105.969392] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8105.977130] Lustre: 187764:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8106.073838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8112.245090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8112.270687] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8112.462317] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8113.230301] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8123.702984] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8129.086061] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8129.276587] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8134.870461] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8135.103987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8138.211519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8138.224973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8138.258695] Lustre: Skipped 2 previous similar messages [ 8138.282520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8138.296265] Lustre: Skipped 1 previous similar message [ 8143.333514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8143.342731] Lustre: Skipped 1 previous similar message [ 8143.573621] Lustre: server umount lustre-OST0000 complete [ 8147.938441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8147.956067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8147.977032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8153.569372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8153.587050] Lustre: Skipped 3 previous similar messages [ 8153.805803] Lustre: server umount lustre-MDT0000 complete [ 8157.331851] LustreError: 185785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786667477 with bad export cookie 12644752089516486427 [ 8157.336697] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8157.339097] LustreError: 185785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8157.631790] Lustre: server umount lustre-MDT0001 complete [ 8167.988625] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 20:31:26 (1786667486) [ 8190.003943] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 8193.573553] Key type lgssc unregistered [ 8193.796620] LNet: 190713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8193.801546] LNetError: 190713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8193.810937] LNet: Removed LNI 192.168.201.159@tcp [ 8194.400609] Key type .llcrypt unregistered [ 8194.402281] Key type ._llcrypt unregistered [ 8216.534271] Key type ._llcrypt registered [ 8216.536045] Key type .llcrypt registered [ 8216.627757] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 8218.104965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8218.144515] alg: No test for adler32 (adler32-zlib) [ 8219.448403] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 8219.715101] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 8221.471161] Key type lgssc registered [ 8223.777962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8236.086829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8236.159763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8238.145166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8244.282783] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8247.274029] LustreError: 191933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8247.300623] LustreError: 191933:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8252.393338] LustreError: 191934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8254.356264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8254.898809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8259.879395] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8267.947921] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8276.107415] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8284.209382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8284.480970] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8284.490535] LustreError: 193798:0:(llog.c:1696:llog_backup()) MGC192.168.201.159@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8284.500318] Lustre: 193798:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.159@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8284.677195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8291.200968] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8299.510786] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8305.122901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8305.124670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8305.150437] Lustre: Skipped 1 previous similar message [ 8305.168853] Lustre: Skipped 2 previous similar messages [ 8308.521788] Lustre: server umount lustre-MDT0000 complete [ 8310.252594] LustreError: 191934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8310.275955] LustreError: 191934:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8313.309348] LustreError: 191915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786667633 with bad export cookie 14537227474045337567 [ 8313.332118] LustreError: 191915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8313.332642] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8313.697406] Lustre: server umount lustre-MDT0001 complete [ 8329.113997] Lustre: server umount lustre-OST0000 complete [ 8339.735293] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 8353.477436] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 8403.411186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8403.765283] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8403.806993] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8403.911188] Lustre: lustre-MDT0000: new disk, initializing [ 8403.999409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8404.030581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8410.439857] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8422.607495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8422.751507] Lustre: 197326:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8422.795754] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8422.798223] Lustre: Skipped 1 previous similar message [ 8422.804555] LustreError: 198112:0:(llog.c:1696:llog_backup()) MGC192.168.201.159@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8422.811498] LustreError: 198112:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8422.816545] Lustre: 198112:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.159@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8422.844844] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8422.850949] Lustre: Skipped 1 previous similar message [ 8422.940976] Lustre: lustre-MDT0001: new disk, initializing [ 8423.048155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8423.086795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8423.094778] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8427.445700] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8438.344710] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8445.808293] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8455.291484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8455.630050] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8455.632818] Lustre: Skipped 1 previous similar message [ 8455.635299] LustreError: 199274:0:(llog.c:1696:llog_backup()) MGC192.168.201.159@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8455.640769] LustreError: 199274:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8455.647805] Lustre: 199274:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.159@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8455.655134] Lustre: 199274:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8455.673359] Lustre: lustre-OST0000: new disk, initializing [ 8455.676377] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8455.684232] Lustre: 199297:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8455.791752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8462.840663] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8465.494061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8465.506482] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8465.614543] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8475.383132] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8481.732891] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8481.912368] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8486.305778] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8486.485716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8491.497867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8491.505983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8491.515225] LustreError: Skipped 1 previous similar message [ 8491.538714] Lustre: Skipped 1 previous similar message [ 8491.550046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8491.572097] Lustre: Skipped 1 previous similar message [ 8494.384516] Lustre: server umount lustre-OST0000 complete [ 8500.192218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8500.197641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8500.208162] Lustre: Skipped 1 previous similar message [ 8500.225953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8504.498791] Lustre: server umount lustre-MDT0000 complete [ 8506.854878] LustreError: 197338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8506.880853] LustreError: 197338:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8508.636920] LustreError: 197317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786667828 with bad export cookie 14537227474045338274 [ 8508.647119] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8508.648809] LustreError: 197317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8508.990617] Lustre: server umount lustre-MDT0001 complete [ 8517.186581] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 20:37:16 (1786667836) [ 8536.997180] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 8547.340126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8547.789903] LustreError: 202426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8547.854624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8551.719111] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8558.055039] LustreError: 202427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8558.081029] LustreError: 202427:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8561.060277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8566.702371] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8569.802814] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8576.599527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8583.381477] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8593.341422] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8593.562629] Lustre: lustre-OST0001: new disk, initializing [ 8593.566040] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8593.570581] Lustre: 204733:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8593.659897] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8593.667244] Lustre: Skipped 2 previous similar messages [ 8599.598251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8599.619410] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8599.720661] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8599.889361] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8611.185760] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8621.229286] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8623.685939] Lustre: DEBUG MARKER: ADD OST3 [ 8629.833401] loop: module loaded [ 8629.967403] loop0: detected capacity change from 0 to 204800000 [ 8630.248327] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8630.268705] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8630.290815] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8636.243927] loop0: detected capacity change from 0 to 204800000 [ 8637.363347] Lustre: DEBUG MARKER: STOP OST3 [ 8642.265267] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8642.482834] Lustre: lustre-OST0003: new disk, initializing [ 8642.495240] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8642.512064] Lustre: 206714:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8644.033141] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8644.234877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8644.245369] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8644.295641] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8646.048232] LustreError: 206845:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8646.930112] Lustre: Failing over lustre-OST0003 [ 8647.121444] Lustre: server umount lustre-OST0003 complete [ 8649.196465] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8649.204362] Lustre: Skipped 2 previous similar messages [ 8649.207209] LustreError: 203910:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8649.225764] LustreError: 203910:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8649.931326] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8659.300287] Lustre: DEBUG MARKER: START OST3 again [ 8667.285639] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8667.688550] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8667.695089] Lustre: Skipped 1 previous similar message [ 8667.715795] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8669.163690] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8669.225708] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8669.226421] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8675.818599] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8702.154573] Lustre: Failing over lustre-OST0003 [ 8702.435551] Lustre: server umount lustre-OST0003 complete [ 8703.505574] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8703.520546] LustreError: 203910:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8703.536886] Lustre: Skipped 1 previous similar message [ 8703.579143] LustreError: 203910:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 8725.509259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8725.513974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8725.523718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8725.528891] Lustre: Skipped 2 previous similar messages [ 8731.967564] Lustre: server umount lustre-MDT0000 complete [ 8735.877633] LustreError: 202408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786668055 with bad export cookie 14537227474045339163 [ 8735.900970] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8735.904363] LustreError: 202408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8739.305803] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8739.314833] Lustre: Skipped 5 previous similar messages [ 8742.720817] Lustre: server umount lustre-MDT0001 complete [ 8748.043229] Lustre: server umount lustre-OST0000 complete [ 8753.808833] Lustre: server umount lustre-OST0001 complete [ 8762.864420] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 8777.039128] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 8820.045073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8820.373902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8820.428892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8820.524405] Lustre: lustre-MDT0000: new disk, initializing [ 8820.659514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8820.699470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8825.718705] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8838.944777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8839.174145] Lustre: 212197:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8839.349297] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8839.353555] Lustre: Skipped 1 previous similar message [ 8839.455364] Lustre: lustre-MDT0001: new disk, initializing [ 8839.664847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8839.678443] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8844.649532] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8854.794268] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8861.823696] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8870.423679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8870.818879] Lustre: 214170:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8872.594554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8872.629697] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8872.965573] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8879.925917] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8890.266802] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8895.404609] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8895.584413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8900.640491] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8900.837338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8903.657959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8903.661504] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8903.688357] Lustre: Skipped 5 previous similar messages [ 8903.697106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8903.701696] Lustre: Skipped 2 previous similar messages [ 8908.791765] Lustre: server umount lustre-OST0000 complete [ 8918.222774] Lustre: server umount lustre-MDT0000 complete [ 8919.010477] LustreError: 212211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8919.021232] LustreError: 212211:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 8922.340633] LustreError: 212191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786668242 with bad export cookie 14537227474045341459 [ 8922.349072] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8922.349649] LustreError: 212191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8922.907609] Lustre: server umount lustre-MDT0001 complete [ 8927.896391] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 20:44:05 (1786668245) [ 8947.658239] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 8958.519082] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 8998.717855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8998.950533] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8998.977967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8998.982843] Lustre: Skipped 1 previous similar message [ 8999.031288] Lustre: lustre-MDT0000: new disk, initializing [ 8999.033015] Lustre: Skipped 1 previous similar message [ 8999.095867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9004.264434] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9017.570550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9017.818827] Lustre: 218783:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9018.379654] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9025.809193] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9035.482444] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9043.031611] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9054.139650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9054.435290] Lustre: 220754:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9056.205373] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9056.488860] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9067.119934] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9083.262356] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9090.188228] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9090.412370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9094.671615] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9094.827666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9097.185839] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9097.216368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9097.220139] Lustre: Skipped 4 previous similar messages [ 9097.251641] Lustre: Skipped 3 previous similar messages [ 9103.586991] Lustre: server umount lustre-OST0000 complete [ 9110.496468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9110.501525] LustreError: Skipped 1 previous similar message [ 9115.038565] Lustre: server umount lustre-MDT0000 complete [ 9117.664963] LustreError: 218794:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9117.697932] LustreError: 218794:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9119.854473] LustreError: 220753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786668439 with bad export cookie 14537227474045342341 [ 9119.870646] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9119.876818] LustreError: 220753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9120.664205] Lustre: server umount lustre-MDT0001 complete [ 9131.405684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9131.831164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9131.835837] Lustre: Skipped 5 previous similar messages [ 9138.657993] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9148.981043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9155.395915] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9164.358639] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9172.131709] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9179.211174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9186.191525] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9194.440284] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9212.399058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9212.418319] Lustre: Skipped 2 previous similar messages [ 9212.430228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9212.436421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9212.444565] Lustre: Skipped 6 previous similar messages [ 9217.025491] Lustre: server umount lustre-OST0000 complete [ 9227.741832] Lustre: server umount lustre-MDT0000 complete [ 9233.381413] LustreError: 222956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786668553 with bad export cookie 14537227474045343223 [ 9233.383577] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9233.397695] LustreError: 222956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9234.045444] Lustre: server umount lustre-MDT0001 complete [ 9337.565891] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 9340.879990] Key type lgssc unregistered [ 9341.227343] LNet: 226662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9341.242775] LNetError: 226662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9341.288365] LNet: Removed LNI 192.168.201.159@tcp [ 9342.336496] Key type .llcrypt unregistered [ 9342.342063] Key type ._llcrypt unregistered [ 9372.588845] Key type ._llcrypt registered [ 9372.598192] Key type .llcrypt registered [ 9372.795358] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 9396.218358] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 9397.493868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9397.545142] alg: No test for adler32 (adler32-zlib) [ 9398.599779] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 9398.945039] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 9400.664428] Key type lgssc registered [ 9402.380507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9439.177725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9439.198680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9440.642645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9440.701522] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9440.802748] Lustre: lustre-MDT0000: new disk, initializing [ 9440.885373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9440.934087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9446.157894] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9456.884830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9456.970196] Lustre: 230018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9457.020504] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9457.024834] Lustre: Skipped 1 previous similar message [ 9457.062238] Lustre: lustre-MDT0001: new disk, initializing [ 9457.109698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9457.156333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9457.173730] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9461.634845] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9473.387382] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9481.395301] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9489.236811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9489.653175] Lustre: lustre-OST0000: new disk, initializing [ 9489.656926] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9489.668635] Lustre: 231987:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9489.784098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9495.610390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9495.625239] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9495.804841] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9496.504723] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9505.868836] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9510.360825] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9510.573149] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9514.683751] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9514.836893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9521.645819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9521.652515] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9521.654014] LustreError: Skipped 1 previous similar message [ 9521.657075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9521.675457] Lustre: Skipped 1 previous similar message [ 9521.690534] Lustre: Skipped 1 previous similar message [ 9526.762660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9526.774039] Lustre: Skipped 1 previous similar message [ 9531.359162] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9531.646867] Lustre: server umount lustre-OST0000 complete [ 9536.995345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9537.033826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9537.045258] Lustre: Skipped 1 previous similar message [ 9542.114704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9542.118989] Lustre: Skipped 1 previous similar message [ 9542.942349] Lustre: server umount lustre-MDT0000 complete [ 9547.575916] LustreError: 230056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786668867 with bad export cookie 1379715209086375704 [ 9547.578442] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9547.587618] LustreError: 230056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9548.004795] Lustre: server umount lustre-MDT0001 complete [ 9574.718535] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 9586.268682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9586.881822] LustreError: 234920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9586.913249] LustreError: 234920:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9587.021194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9592.290745] LustreError: 234921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9592.905537] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9596.386385] LustreError: 234920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9601.510840] LustreError: 234920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9604.675117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9605.150784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9613.017871] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9617.043053] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9625.453325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9626.271848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9633.267814] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9642.399154] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9642.557413] Lustre: lustre-OST0001: new disk, initializing [ 9642.560759] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9642.565812] Lustre: 237230:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9642.654912] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9649.347603] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9652.919357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9652.933980] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9653.071751] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9664.349135] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9673.909814] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9685.306824] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 20:56:44 (1786669004) [ 9689.061521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9689.062921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9689.073631] Lustre: Skipped 3 previous similar messages [ 9689.078457] Lustre: Skipped 1 previous similar message [ 9693.910428] Lustre: server umount lustre-MDT0000 complete [ 9694.187532] LustreError: 236121:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9694.225316] LustreError: 236121:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9698.202538] LustreError: 234901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786669018 with bad export cookie 1379715209086376586 [ 9698.206945] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9698.213418] LustreError: 234901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9698.707812] Lustre: server umount lustre-MDT0001 complete [ 9712.728075] Lustre: server umount lustre-OST0000 complete [ 9715.681775] Lustre: 228242:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786669019/real 1786669019] req@ffff8a2343564380 x1873457937763584/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786669035 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9715.725524] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9715.747196] Lustre: Skipped 1 previous similar message [ 9716.818202] Lustre: server umount lustre-OST0001 complete [ 9724.743112] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 9742.915224] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 9791.807620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9792.254725] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9792.298359] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9792.450966] Lustre: lustre-MDT0000: new disk, initializing [ 9792.574487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9792.600814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9798.242984] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9813.217682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9813.377911] Lustre: 241200:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9813.450310] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9813.455938] Lustre: Skipped 1 previous similar message [ 9813.585349] Lustre: lustre-MDT0001: new disk, initializing [ 9813.755853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9813.771749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9818.396840] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9828.921416] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9836.158918] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9845.036633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9845.503432] Lustre: lustre-OST0000: new disk, initializing [ 9845.516995] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9845.532396] Lustre: 243166:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9847.186362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9847.215704] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9847.506699] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9854.630117] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9867.906560] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9874.851838] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9875.282493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9881.409545] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9881.596085] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9888.229477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9888.232621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9888.260323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9888.260332] Lustre: Skipped 2 previous similar messages [ 9890.459810] Lustre: server umount lustre-OST0000 complete [ 9895.913984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9898.474614] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9898.484712] Lustre: Skipped 2 previous similar messages [ 9901.603398] Lustre: server umount lustre-MDT0000 complete [ 9903.587942] LustreError: 241210:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9903.611753] LustreError: 241210:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9906.459769] LustreError: 241191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786669226 with bad export cookie 1379715209086378714 [ 9906.468289] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9906.477607] LustreError: 241191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9907.150483] Lustre: server umount lustre-MDT0001 complete [ 9918.146653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9918.828695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9918.831708] Lustre: Skipped 2 previous similar messages [ 9924.091684] LustreError: 245386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9924.114772] LustreError: 245386:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9924.870988] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9935.455389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9940.522304] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9950.474390] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9957.804503] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9968.339745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9978.426208] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9988.811482] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10133.988380] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10133.996464] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10134.005165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10134.008105] Lustre: Skipped 4 previous similar messages [10139.973946] Lustre: server umount lustre-OST0000 complete [10145.764834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10145.772053] LustreError: Skipped 1 previous similar message [10150.212251] Lustre: server umount lustre-MDT0000 complete [10152.417527] LustreError: 245382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10152.451862] LustreError: 245382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10154.851203] LustreError: 245366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786669474 with bad export cookie 1379715209086379603 [10154.855395] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10154.865617] LustreError: 245366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10155.271653] Lustre: server umount lustre-MDT0001 complete [10257.787961] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [10260.372229] Key type lgssc unregistered [10260.736821] LNet: 249011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10260.747221] LNetError: 249011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10260.769513] LNet: Removed LNI 192.168.201.159@tcp [10261.700169] Key type .llcrypt unregistered [10261.706660] Key type ._llcrypt unregistered [10293.600905] Key type ._llcrypt registered [10293.603599] Key type .llcrypt registered [10293.748028] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [10308.681451] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [10309.858389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10310.018697] alg: No test for adler32 (adler32-zlib) [10311.238537] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [10311.507226] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [10313.248198] Key type lgssc registered [10314.549962] Lustre: Echo OBD driver; http://www.lustre.org/ [10369.862788] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [10381.055687] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10381.091846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10382.438754] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10382.494864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10382.649343] Lustre: lustre-MDT0000: new disk, initializing [10382.841569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10382.886860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10387.855535] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10399.223487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10399.320777] Lustre: 253084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10399.366254] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10399.370091] Lustre: Skipped 1 previous similar message [10399.429576] Lustre: lustre-MDT0001: new disk, initializing [10399.499038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10399.530547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10399.536074] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10404.184785] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10409.169972] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10416.999894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10417.557198] Lustre: lustre-OST0000: new disk, initializing [10417.579615] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10417.588783] Lustre: 254715:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10417.754892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10418.699173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10418.706385] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10418.810874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10426.042621] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10441.495084] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10441.891785] Lustre: lustre-OST0001: new disk, initializing [10441.909658] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10441.925426] Lustre: 255589:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10442.100947] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10450.096213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10450.104875] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10450.279507] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10450.303899] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10465.587720] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10474.452262] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10486.755095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10486.762480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10486.771446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10491.360190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10491.362224] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10491.372693] Lustre: Skipped 2 previous similar messages [10491.395115] Lustre: Skipped 2 previous similar messages [10493.079971] Lustre: server umount lustre-MDT0000 complete [10497.863869] LustreError: 253077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786669817 with bad export cookie 9589308819432644843 [10497.869247] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10497.878566] LustreError: 253077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10498.470201] Lustre: server umount lustre-MDT0001 complete [10513.635556] Lustre: server umount lustre-OST0000 complete [10528.217395] Lustre: server umount lustre-OST0001 complete [10544.941690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10557.436483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10572.457458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10586.461756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10610.673230] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [10621.059653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10621.078754] Lustre: lustre-MDT0000: reset Object Index mappings [10621.689621] LustreError: 260600:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10621.713777] LustreError: 260600:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10621.763826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10627.039720] LustreError: 260601:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10627.844466] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10632.166328] LustreError: 260600:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10637.283753] LustreError: 260601:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10640.679578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10640.712104] Lustre: lustre-MDT0001: reset Object Index mappings [10641.282215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10649.076587] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10653.375401] Lustre: 261746:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10661.730762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10662.067552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10666.151783] LustreError: 262101:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10667.217125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10668.086714] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10676.975645] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10682.366823] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10683.697568] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10694.015620] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10700.074465] Lustre: 263618:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10707.936877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10707.946187] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10707.959480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10707.959488] Lustre: Skipped 1 previous similar message [10708.031983] Lustre: Skipped 3 previous similar messages [10712.777769] Lustre: server umount lustre-MDT0000 complete [10713.059233] LustreError: 261336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10713.072531] LustreError: 261336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10717.280903] LustreError: 261347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786670037 with bad export cookie 9589308819432651633 [10717.294192] LustreError: 261347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10717.294237] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10717.671852] Lustre: server umount lustre-MDT0001 complete [10732.754244] Lustre: server umount lustre-OST0000 complete [10733.348772] Lustre: 250591:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786670037/real 1786670037] req@ffff8a247f010a80 x1873458895133184/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786670053 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10733.393137] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10736.782665] Lustre: server umount lustre-OST0001 complete [10757.878934] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 21:14:36 (1786670076) [10777.239039] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [10790.494858] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [10835.388727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10835.705996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10835.761805] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10835.867652] Lustre: lustre-MDT0000: new disk, initializing [10835.977251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10835.987249] Lustre: Skipped 1 previous similar message [10835.999808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10842.145615] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10854.235780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10854.364353] Lustre: 267876:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10854.486649] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10854.490422] Lustre: Skipped 1 previous similar message [10854.604894] Lustre: lustre-MDT0001: new disk, initializing [10854.773593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10854.801648] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10860.134865] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10872.501584] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10881.281072] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10891.881779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10892.274437] Lustre: lustre-OST0000: new disk, initializing [10892.281610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10892.287922] Lustre: 269845:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10893.823485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10893.829968] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10893.953236] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10899.545661] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10910.258428] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10915.211730] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10915.448546] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10920.562469] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10920.791439] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10924.521681] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10924.523466] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10924.536284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10924.536292] Lustre: Skipped 3 previous similar messages [10924.568000] Lustre: Skipped 1 previous similar message [10929.643698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10929.657373] Lustre: Skipped 1 previous similar message [10937.311308] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10937.542133] Lustre: server umount lustre-OST0000 complete [10941.921536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10941.928245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10941.956161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10941.967570] Lustre: Skipped 3 previous similar messages [10956.255222] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10956.498653] Lustre: server umount lustre-MDT0000 complete [10960.366767] LustreError: 269257:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10960.383830] LustreError: 269257:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10961.368602] LustreError: 268669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786670281 with bad export cookie 9589308819432653726 [10961.370116] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10961.378782] LustreError: 268669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10961.906360] Lustre: server umount lustre-MDT0001 complete [10986.400752] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [11001.163631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11001.972692] LustreError: 272784:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11002.067976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11002.086485] Lustre: Skipped 2 previous similar messages [11007.356318] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11017.771111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11024.242149] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11027.393540] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11034.877226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11044.948139] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11056.705968] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11056.956830] Lustre: lustre-OST0001: new disk, initializing [11056.966153] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11056.976845] Lustre: 275093:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11058.339422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11058.353534] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11058.586421] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11065.313980] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11076.383799] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11080.642735] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11087.077488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11087.133389] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11095.009582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11095.014136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11095.025536] Lustre: Skipped 1 previous similar message [11095.030153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11095.032877] Lustre: Skipped 6 previous similar messages [11101.179687] Lustre: server umount lustre-MDT0000 complete [11104.739179] LustreError: 272781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11104.751870] LustreError: 272781:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [11105.699891] LustreError: 272764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786670425 with bad export cookie 9589308819432654608 [11105.724575] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11105.728316] LustreError: 272764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11106.588776] Lustre: server umount lustre-MDT0001 complete [11124.720713] Lustre: server umount lustre-OST0000 complete [11125.728237] Lustre: 250591:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786670429/real 1786670429] req@ffff8a234ce7df80 x1873458895249536/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786670445 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11126.756253] Lustre: 250589:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786670430/real 1786670430] req@ffff8a246cb19500 x1873458895249792/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786670446 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11129.628205] Lustre: server umount lustre-OST0001 complete [11151.641857] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [11162.867084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11163.473710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11163.483765] Lustre: Skipped 3 previous similar messages [11168.284276] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11177.550880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11182.648318] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11185.824385] Lustre: 279290:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11192.724847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11198.951720] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11207.688845] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11209.115141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11209.209414] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11214.555923] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11223.161142] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11227.498993] Lustre: 281167:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11234.055765] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11234.655593] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11247.684720] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 21:22:45 (1786670565) [11265.009925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11265.015958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11265.035270] Lustre: Skipped 4 previous similar messages [11265.042499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11265.046415] Lustre: Skipped 4 previous similar messages [11270.530494] Lustre: server umount lustre-MDT0000 complete [11270.629272] LustreError: 281620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11270.659373] LustreError: 281620:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [11274.896279] LustreError: 278132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786670594 with bad export cookie 9589308819432660936 [11274.899257] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11274.906026] LustreError: 278132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [11281.515774] Lustre: server umount lustre-MDT0001 complete [11291.405423] Lustre: server umount lustre-OST0000 complete [11295.472691] Lustre: server umount lustre-OST0001 complete [11303.616421] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [11315.672907] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [11375.580082] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [11387.192125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11387.515789] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11387.540209] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11387.628625] Lustre: lustre-MDT0000: new disk, initializing [11387.790486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11392.299802] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11402.415568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11402.525941] Lustre: 285435:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11402.792360] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11408.475794] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11421.325907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11421.557218] Lustre: 287067:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11421.670155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11421.675599] Lustre: Skipped 5 previous similar messages [11423.484900] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11429.000275] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11440.483392] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11440.680235] Lustre: 287936:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11442.349856] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11442.359339] Lustre: Skipped 1 previous similar message [11442.595805] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11447.729651] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11458.778126] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11463.517498] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11463.528128] Lustre: Skipped 1 previous similar message [11484.566336] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 21:26:42 (1786670802) [11487.518685] Lustre: server umount lustre-MDT0000 complete [11488.747106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11488.772870] Lustre: Skipped 5 previous similar messages [11491.637757] LustreError: 285427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786670811 with bad export cookie 9589308819432668825 [11491.638790] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11491.646052] LustreError: 285427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11491.952295] Lustre: server umount lustre-MDT0001 complete [11507.029672] Lustre: server umount lustre-OST0000 complete [11509.867928] Lustre: 250591:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786670813/real 1786670813] req@ffff8a234cef2300 x1873458895403648/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786670829 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11510.866401] Lustre: server umount lustre-OST0001 complete [11519.516084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11524.739657] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11529.195814] LustreError: 290183:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11529.215270] LustreError: 290183:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [11533.091191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11538.503971] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11547.062370] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11554.440961] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11561.437487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11568.182173] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11576.496273] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11596.010471] loop1: detected capacity change from 0 to 204800000 [11596.244628] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11596.262248] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11596.280415] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11601.968087] loop1: detected capacity change from 0 to 204800000 [11606.991378] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11607.093786] LustreError: 290171:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs5218: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11607.117163] LustreError: 290171:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs5218 [11607.123246] LustreError: 290171:0:(mgs_handler.c:517:mgs_target_reg()) fs5218-MDT0001: Error checking target index: rc = -13 [11607.133453] LustreError: fs5218-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11607.141978] LustreError: 293901:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11607.148943] LustreError: 293901:0:(tgt_mount.c:2129:server_put_super()) no obd fs5218-MDT0001 [11607.153428] LustreError: 293901:0:(tgt_mount.c:116:server_deregister_mount()) fs5218-MDT0001 not registered [11607.201874] Lustre: server umount fs5218-MDT0001 complete [11607.205554] LustreError: 293901:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11615.955379] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11616.098594] Lustre: Setting parameter fs5218-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs5218-MDT0001 [11616.190717] Lustre: srv-fs5218-MDT0001: No data found on store. Initialize space. [11616.195823] Lustre: Skipped 4 previous similar messages [11616.292536] Lustre: fs5218-MDT0001: new disk, initializing [11616.295370] Lustre: Skipped 3 previous similar messages [11616.506345] Lustre: srv-fs5218-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11616.524107] Lustre: 294369:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5218-MDT0001-mdtlov: can't create new llog: rc = -115 [11616.539435] LustreError: 294369:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs5218-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11621.067313] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11628.263090] Lustre: server umount fs5218-MDT0001 complete [11642.262869] loop1: detected capacity change from 0 to 204800000 [11644.922631] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11648.233198] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11648.570611] Lustre: 295937:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5218-MDT0001-mdtlov: can't create new llog: rc = -115 [11648.582919] LustreError: 295937:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs5218-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11654.385374] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11659.562794] Lustre: server umount fs5218-MDT0001 complete [11670.500556] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11670.505424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11670.516549] Lustre: Skipped 5 previous similar messages [11670.521603] Lustre: Skipped 9 previous similar messages [11675.651475] Lustre: server umount lustre-MDT0000 complete [11683.516930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11683.627676] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11688.774843] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11689.450079] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11696.434288] loop1: detected capacity change from 0 to 204800000 [11703.155625] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11703.870996] Lustre: 298383:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5218-MDT0001-mdtlov: can't create new llog: rc = -115 [11703.884627] LustreError: 298383:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs5218-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11709.124116] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11714.132667] Lustre: server umount fs5218-MDT0001 complete [11740.131352] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11740.139055] LustreError: Skipped 1 previous similar message [11743.620983] Lustre: server umount lustre-OST0000 complete [11754.255769] Lustre: server umount lustre-MDT0000 complete [11759.079547] LustreError: 290162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786671078 with bad export cookie 9589308819432671506 [11759.098336] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11759.594906] Lustre: server umount lustre-MDT0001 complete [11775.854529] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [11780.721720] Key type lgssc unregistered [11781.391200] LNet: 300733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11781.408903] LNetError: 300733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11781.425855] LNet: Removed LNI 192.168.201.159@tcp [11783.655586] Key type .llcrypt unregistered [11783.665144] Key type ._llcrypt unregistered [11810.936335] Key type ._llcrypt registered [11810.938381] Key type .llcrypt registered [11811.052745] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [11812.077327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11812.155400] alg: No test for adler32 (adler32-zlib) [11813.168262] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [11813.366678] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [11815.016430] Key type lgssc registered [11816.181415] Lustre: Echo OBD driver; http://www.lustre.org/ [11824.812472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11824.837863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11826.374150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11830.900808] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11836.898930] LustreError: 302007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11836.911395] LustreError: 302007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11839.994971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11840.444728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11844.577082] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11851.374926] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11857.922565] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11864.991071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11865.472557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11866.544379] LustreError: 303835:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11871.670802] LustreError: 303836:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11871.696314] LustreError: 303836:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11871.940201] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11876.838072] LustreError: 303837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11876.861915] LustreError: 303837:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11880.782854] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11881.970884] LustreError: 303837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11881.987899] LustreError: 303837:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11892.204137] LustreError: 303836:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11892.225885] LustreError: 303836:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11901.923162] loop1: detected capacity change from 0 to 204800000 [11902.075457] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11902.090475] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11902.101749] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11908.124081] loop1: detected capacity change from 0 to 204800000 [11909.379410] LustreError: 303836:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11909.400486] LustreError: 303836:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [11915.239538] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11915.399336] Lustre: fs8221-OST0000: new disk, initializing [11915.404266] Lustre: srv-fs8221-OST0000: No data found on store. Initialize space. [11915.428800] Lustre: 305743:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11915.527310] Lustre: fs8221-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11925.895029] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11936.844980] Lustre: server umount fs8221-OST0000 complete [11943.397634] LustreError: 303837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11943.435670] LustreError: 303837:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [11949.970316] loop1: detected capacity change from 0 to 204800000 [11953.655935] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11956.336157] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11956.708384] Lustre: fs8221-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11964.491894] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11977.953118] loop2: detected capacity change from 0 to 204800000 [11978.386886] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11978.406795] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11978.424572] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11985.898237] loop2: detected capacity change from 0 to 204800000 [11993.086483] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11993.223409] LustreError: fs8221-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11993.239344] LustreError: fs8221-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11993.248427] LustreError: 308786:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11993.260544] LustreError: 308786:0:(tgt_mount.c:2129:server_put_super()) no obd fs8221-OST0001 [11993.266606] LustreError: 308786:0:(tgt_mount.c:116:server_deregister_mount()) fs8221-OST0001 not registered [11993.308924] Lustre: server umount fs8221-OST0001 complete [11993.311710] LustreError: 308786:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [12005.231791] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12005.426583] Lustre: fs8221-OST0001: new disk, initializing [12005.433190] Lustre: srv-fs8221-OST0001: No data found on store. Initialize space. [12005.449979] Lustre: 309337:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12005.582625] Lustre: fs8221-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12009.971946] LustreError: 303906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12009.999512] LustreError: 303906:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [12012.151539] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [12022.740953] Lustre: server umount fs8221-OST0000 complete [12037.886059] Lustre: server umount fs8221-OST0001 complete [12053.770277] loop1: detected capacity change from 0 to 204800000 [12060.463773] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12060.887318] Lustre: fs8221-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12067.840812] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [12075.852690] loop2: detected capacity change from 0 to 204800000 [12083.095657] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12091.208629] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [12104.034628] Lustre: server umount fs8221-OST0000 complete [12107.241165] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12107.245767] LustreError: Skipped 1 previous similar message [12115.788167] Lustre: server umount fs8221-OST0001 complete [12132.833231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12132.839168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12132.850949] Lustre: Skipped 1 previous similar message [12132.862477] Lustre: Skipped 1 previous similar message [12137.962619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12137.979786] LustreError: 303836:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12137.999373] LustreError: 303836:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 73 previous similar messages [12143.082609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12143.093544] Lustre: Skipped 2 previous similar messages [12146.143437] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12146.371044] Lustre: server umount lustre-OST0000 complete [12152.803597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12152.812052] LustreError: Skipped 1 previous similar message [12152.814072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12152.826694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12157.596984] Lustre: server umount lustre-MDT0000 complete [12162.259841] LustreError: 301988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786671482 with bad export cookie 288172545804038339 [12162.273445] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12162.277416] LustreError: 301988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12162.729822] Lustre: server umount lustre-MDT0001 complete [12175.207716] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [12178.637316] Key type lgssc unregistered [12179.057242] LNet: 315270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12179.065601] LNetError: 315270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12179.096862] LNet: Removed LNI 192.168.201.159@tcp [12180.098397] Key type .llcrypt unregistered [12180.101048] Key type ._llcrypt unregistered [12221.272708] Key type ._llcrypt registered [12221.278514] Key type .llcrypt registered [12221.403037] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [12239.844469] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [12241.060186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12241.231421] alg: No test for adler32 (adler32-zlib) [12242.441255] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [12242.736872] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [12244.447242] Key type lgssc registered [12245.692643] Lustre: Echo OBD driver; http://www.lustre.org/ [12291.097566] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12291.148949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12292.682733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12292.739796] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12292.929993] Lustre: lustre-MDT0000: new disk, initializing [12293.064598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12293.090531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12298.872278] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [12310.664639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12310.825279] Lustre: 319134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12310.915933] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12310.923224] Lustre: Skipped 1 previous similar message [12311.039584] Lustre: lustre-MDT0001: new disk, initializing [12311.246460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12311.340235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12311.356529] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12317.268586] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [12327.962666] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12335.209034] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12342.828756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12343.151618] Lustre: lustre-OST0000: new disk, initializing [12343.167173] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12343.180752] Lustre: 321104:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12343.314329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12349.035559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12349.047756] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12349.246111] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12350.091522] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [12360.744911] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12363.602856] Lustre: Mounted lustre-client - version 2.17.57_1_gb6c95b3 [12364.953467] Lustre: Unmounted lustre-client [12379.115321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12379.153140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12384.227409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12384.253609] Lustre: Skipped 2 previous similar messages [12389.363358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12389.372938] Lustre: Skipped 1 previous similar message [12390.367170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12390.789770] Lustre: server umount lustre-OST0000 complete [12398.560826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12398.570598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12398.593794] Lustre: Skipped 1 previous similar message [12398.606361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12399.585975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12404.707703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12404.717932] Lustre: Skipped 3 previous similar messages [12411.359156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12412.044771] Lustre: server umount lustre-MDT0000 complete [12416.551754] LustreError: 319126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786671736 with bad export cookie 1066970972811708185 [12416.580502] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12417.108519] Lustre: server umount lustre-MDT0001 complete [12424.095354] LNet: 1 local NIs in recovery (showing 1): 192.168.201.159@tcp [12429.626322] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [12433.491814] Key type lgssc unregistered [12433.771739] LNet: 323274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12433.783571] LNetError: 323274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12494.248882] LNet: Removed LNI 192.168.201.159@tcp [12496.138694] Key type .llcrypt unregistered [12496.146115] Key type ._llcrypt unregistered [12525.114614] Key type ._llcrypt registered [12525.126355] Key type .llcrypt registered [12525.351357] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [12544.801554] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [12547.039656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12547.136510] alg: No test for adler32 (adler32-zlib) [12548.432989] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [12548.771336] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [12550.519847] Key type lgssc registered [12552.501070] Lustre: Echo OBD driver; http://www.lustre.org/ [12595.180719] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 21:45:13 (1786671913) [12603.809198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12603.858933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12605.525029] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12605.577609] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12605.666720] Lustre: lustre-MDT0000: new disk, initializing [12605.758859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12605.781976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12611.113610] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [12626.211559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12626.401852] Lustre: 326500:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12626.482295] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12626.489519] Lustre: Skipped 1 previous similar message [12626.645305] Lustre: lustre-MDT0001: new disk, initializing [12626.755559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12626.818397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12626.830325] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12634.189656] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [12645.467757] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12652.623757] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12661.725984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12662.160288] Lustre: lustre-OST0000: new disk, initializing [12662.164865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12662.175118] Lustre: 328470:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12662.284718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12670.345339] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [12671.139806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12671.153458] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12671.355600] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12682.331438] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12702.176720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12702.203359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12702.203814] LustreError: Skipped 1 previous similar message [12702.204545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12702.262392] Lustre: Skipped 1 previous similar message [12706.748711] Lustre: server umount lustre-OST0000 complete [12712.417034] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12712.422618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12712.438703] Lustre: Skipped 1 previous similar message [12712.467537] Lustre: Skipped 2 previous similar messages [12717.540610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12717.554906] Lustre: Skipped 1 previous similar message [12718.588780] Lustre: server umount lustre-MDT0000 complete [12724.481213] LustreError: 326492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786672044 with bad export cookie 4472930123750547587 [12724.486076] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12724.497305] LustreError: 326492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12724.905298] Lustre: server umount lustre-MDT0001 complete [12741.340622] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [12744.894477] Key type lgssc unregistered [12745.255909] LNet: 330384:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12745.265304] LNetError: 330384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12745.298817] LNet: Removed LNI 192.168.201.159@tcp [12746.052655] Key type .llcrypt unregistered [12746.054787] Key type ._llcrypt unregistered [12773.985025] Key type ._llcrypt registered [12773.991945] Key type .llcrypt registered [12774.132348] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [12775.968059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12776.129486] alg: No test for adler32 (adler32-zlib) [12777.346654] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [12777.738779] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [12779.476817] Key type lgssc registered [12780.808469] Lustre: Echo OBD driver; http://www.lustre.org/ [12789.838770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12789.880954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12791.265851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12795.868354] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [12801.510134] LustreError: 331658:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12801.542178] LustreError: 331658:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12804.752750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12805.120885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12810.367540] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [12819.288936] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12827.925821] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12837.900622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12838.301278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12846.414551] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [12857.247558] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12878.100244] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12881.014686] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12887.009234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12887.021863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12887.039567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12887.528950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12887.542466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12887.565239] Lustre: Skipped 1 previous similar message [12889.928266] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.59@tcp (stopping) [12892.653950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12892.662587] Lustre: Skipped 2 previous similar messages [12897.769912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12897.794068] Lustre: Skipped 2 previous similar messages [12901.343116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12901.653159] Lustre: server umount lustre-MDT0000 complete [12902.882676] LustreError: 332623:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12902.906428] LustreError: 332623:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12905.220524] LustreError: 332616:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12906.787424] LustreError: 331639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786672226 with bad export cookie 11004734568810854084 [12906.817959] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12907.999897] LustreError: 331654:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12908.019541] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12908.044020] Lustre: Skipped 1 previous similar message [12908.048447] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12908.056309] Lustre: Skipped 2 previous similar messages [12915.448392] LustreError: 331655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12915.475217] LustreError: 331655:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12921.312357] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12921.673919] Lustre: server umount lustre-MDT0001 complete [12927.114082] Lustre: server umount lustre-OST0000 complete [12940.933227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12948.288431] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [12952.360945] Lustre: 335867:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.159@tcp [12955.898921] Lustre: server umount MGS complete [12966.525160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12967.212113] LustreError: 336440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12967.251395] LustreError: 336440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12967.338477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12972.557263] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [12982.925045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12983.363922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12989.489377] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [13001.563776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13001.866910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13009.238970] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [13018.002678] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13019.622685] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [13026.147651] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13028.440612] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13034.977390] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13034.991880] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13035.009115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13035.022432] Lustre: Skipped 7 previous similar messages [13039.105640] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13039.115363] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13040.843072] Lustre: server umount lustre-OST0000 complete [13049.314306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13051.445496] Lustre: server umount lustre-MDT0000 complete [13054.440989] LustreError: 336440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13054.453034] LustreError: 336440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [13056.321743] LustreError: 336422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786672376 with bad export cookie 11004734568810854924 [13056.341760] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13056.349825] LustreError: 336422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [13056.844711] Lustre: server umount lustre-MDT0001 complete [13073.913489] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [13077.429747] Key type lgssc unregistered [13077.805924] LNet: 339882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13077.813358] LNetError: 339882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13078.897863] LNet: Removed LNI 192.168.201.159@tcp [13079.751812] Key type .llcrypt unregistered [13079.754552] Key type ._llcrypt unregistered [13099.831496] Key type ._llcrypt registered [13099.833300] Key type .llcrypt registered [13099.975898] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [13118.200088] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [13119.160039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13119.432436] alg: No test for adler32 (adler32-zlib) [13120.504472] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [13120.914395] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [13122.663617] Key type lgssc registered [13124.211903] Lustre: Echo OBD driver; http://www.lustre.org/ [13168.962900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13169.014507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13170.303324] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13170.343647] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13170.426534] Lustre: lustre-MDT0000: new disk, initializing [13170.532523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13170.563696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13174.455867] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [13185.483564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13185.596126] Lustre: 343046:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13185.645493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13185.649058] Lustre: Skipped 1 previous similar message [13185.698680] Lustre: lustre-MDT0001: new disk, initializing [13185.795392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13185.832687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13185.841933] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13191.453951] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [13201.996129] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13208.347147] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13215.815940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13216.264944] Lustre: lustre-OST0000: new disk, initializing [13216.268569] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13216.273804] Lustre: 345016:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13216.382408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13222.495193] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [13223.520241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13223.536904] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13223.645242] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13233.640365] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13238.889354] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13239.119449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13244.556385] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13244.744979] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13249.508501] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13249.510798] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13249.522668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13249.525517] Lustre: Skipped 1 previous similar message [13252.998912] Lustre: server umount lustre-OST0000 complete [13259.749434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13259.778444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13259.786899] Lustre: Skipped 2 previous similar messages [13264.555672] Lustre: server umount lustre-MDT0000 complete [13268.520230] LustreError: 343851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786672588 with bad export cookie 15271166920002032530 [13268.522206] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13268.536127] LustreError: 343851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13269.557107] Lustre: server umount lustre-MDT0001 complete [13280.332444] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 21:56:38 (1786672598) [13299.703468] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [13311.393920] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [13353.091775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13353.280722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13353.312422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13353.386373] Lustre: lustre-MDT0000: new disk, initializing [13353.445775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13353.470840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13358.141318] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [13369.612506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13369.724490] Lustre: 349777:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13369.776727] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13369.781328] Lustre: Skipped 1 previous similar message [13369.888111] Lustre: lustre-MDT0001: new disk, initializing [13369.987513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13370.022676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13370.039823] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13374.598444] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [13384.161308] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13390.839309] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13398.849054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13399.120302] Lustre: lustre-OST0000: new disk, initializing [13399.122575] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13399.125759] Lustre: 351746:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13399.192572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13405.455550] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [13408.884985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13408.905188] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13409.037889] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13415.416507] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13455.335493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13455.345683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13455.367919] Lustre: Skipped 1 previous similar message [13455.376367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13455.386227] Lustre: Skipped 1 previous similar message [13459.798900] Lustre: server umount lustre-OST0000 complete [13465.569360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13465.571591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13465.593114] Lustre: Skipped 2 previous similar messages [13465.611844] Lustre: Skipped 1 previous similar message [13469.653741] Lustre: server umount lustre-MDT0000 complete [13470.692510] LustreError: 349788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13470.730656] LustreError: 349788:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13474.088743] LustreError: 349770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786672793 with bad export cookie 15271166920002033412 [13474.092324] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13474.103847] LustreError: 349770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13474.437700] Lustre: server umount lustre-MDT0001 complete [13487.626387] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [13491.096286] Key type lgssc unregistered [13491.511692] LNet: 353929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13491.520719] LNetError: 353929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13491.541848] LNet: Removed LNI 192.168.201.159@tcp [13492.410746] Key type .llcrypt unregistered [13492.416649] Key type ._llcrypt unregistered [13592.888389] Key type ._llcrypt registered [13592.890957] Key type .llcrypt registered [13592.985353] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [13593.247340] Key type .llcrypt unregistered [13593.250106] Key type ._llcrypt unregistered [13682.419340] Key type ._llcrypt registered [13682.423630] Key type .llcrypt registered [13682.553277] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [13682.899575] Key type .llcrypt unregistered [13682.906101] Key type ._llcrypt unregistered [13716.686644] Key type ._llcrypt registered [13716.689319] Key type .llcrypt registered [13716.765542] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [13717.147324] Key type .llcrypt unregistered [13717.149790] Key type ._llcrypt unregistered [13751.807640] Key type ._llcrypt registered [13751.810227] Key type .llcrypt registered [13751.925825] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [13752.182300] Key type .llcrypt unregistered [13752.184181] Key type ._llcrypt unregistered [13850.014283] Key type ._llcrypt registered [13850.018356] Key type .llcrypt registered [13850.155501] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [13850.475796] Key type .llcrypt unregistered [13850.479535] Key type ._llcrypt unregistered [13882.116603] Key type ._llcrypt registered [13882.117912] Key type .llcrypt registered [13882.375989] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [13882.765556] Key type .llcrypt unregistered [13882.767735] Key type ._llcrypt unregistered [13915.771500] Key type ._llcrypt registered [13915.773661] Key type .llcrypt registered [13915.905229] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [13916.217319] Key type .llcrypt unregistered [13916.219717] Key type ._llcrypt unregistered [13945.069783] Key type ._llcrypt registered [13945.071746] Key type .llcrypt registered [13945.195913] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [13945.586539] Key type .llcrypt unregistered [13945.588216] Key type ._llcrypt unregistered [13973.061399] Key type ._llcrypt registered [13973.063622] Key type .llcrypt registered [13973.171443] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [13973.507570] Key type .llcrypt unregistered [13973.512164] Key type ._llcrypt unregistered [14001.576826] Key type ._llcrypt registered [14001.578985] Key type .llcrypt registered [14001.701789] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [14002.127649] Key type .llcrypt unregistered [14002.131367] Key type ._llcrypt unregistered [14067.288724] Key type ._llcrypt registered [14067.291694] Key type .llcrypt registered [14067.480695] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [14067.805349] Key type .llcrypt unregistered [14067.807760] Key type ._llcrypt unregistered [14103.385715] Key type ._llcrypt registered [14103.388801] Key type .llcrypt registered [14103.500144] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [14103.898286] Key type .llcrypt unregistered [14103.905988] Key type ._llcrypt unregistered [14134.575751] Key type ._llcrypt registered [14134.577996] Key type .llcrypt registered [14134.676576] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [14135.097697] Key type .llcrypt unregistered [14135.101158] Key type ._llcrypt unregistered [14190.189333] Key type ._llcrypt registered [14190.191703] Key type .llcrypt registered [14190.289567] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [14205.770264] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [14206.713631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14206.838603] alg: No test for adler32 (adler32-zlib) [14207.856562] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [14208.047748] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [14209.687195] Key type lgssc registered [14210.934503] Lustre: Echo OBD driver; http://www.lustre.org/ [14246.607084] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14246.662639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14247.885701] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14247.920709] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14247.976484] Lustre: tst250fs-MDT0000: new disk, initializing [14248.044892] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14248.060818] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14252.085820] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [14261.087647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14261.209249] Lustre: 374320:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14261.265631] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14261.273502] Lustre: Skipped 1 previous similar message [14261.329806] Lustre: tst250fs-MDT0001: new disk, initializing [14261.428466] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14261.461593] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14261.478495] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14266.026816] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [14271.048661] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14278.210566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14278.434898] Lustre: tst250fs-OST0000: new disk, initializing [14278.438214] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14278.442405] Lustre: 375950:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14278.518070] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14279.262524] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14279.279494] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14279.438547] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14286.009877] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [14297.131569] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14297.288917] Lustre: tst250fs-OST0001: new disk, initializing [14297.292354] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14297.296707] Lustre: 376823:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14297.400692] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14303.371542] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14303.385825] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14303.515451] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14303.907995] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [14309.366657] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14312.249811] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [14312.256919] Lustre: Skipped 2 previous similar messages [14333.412193] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14333.419827] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14333.444665] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14334.432699] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14334.437196] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14334.446527] Lustre: Skipped 1 previous similar message [14334.470732] Lustre: Skipped 1 previous similar message [14337.477132] Lustre: server umount tst250fs-MDT0000 complete [14341.908082] LustreError: 376822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786673661 with bad export cookie 10234032539865264385 [14341.911492] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14341.918664] LustreError: 376822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14342.487181] Lustre: server umount tst250fs-MDT0001 complete [14358.245662] Lustre: server umount tst250fs-OST0000 complete [14360.801610] Lustre: 372544:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786673664/real 1786673664] req@ffff8a234ca27480 x1873462980320640/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786673680 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14360.854535] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14360.878124] Lustre: Skipped 1 previous similar message [14363.103868] Lustre: 372545:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786673666/real 1786673666] req@ffff8a2475f8e680 x1873462980320896/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786673682 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14364.085823] Lustre: server umount tst250fs-OST0001 complete [14390.065770] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [14401.633582] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [14439.146818] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 22:15:57 (1786673757) [14442.381251] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14463.838481] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [14477.063996] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [14511.090337] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14164 sec ========= 22:17:09 (1786673829) [14513.196126] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:17:11 (1786673831) === [14516.704493] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:17:15 (1786673835) === [14563.732088] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [14566.640525] Key type lgssc unregistered [14566.983807] LNet: 385979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14566.997805] LNetError: 385979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14567.009611] LNet: Removed LNI 192.168.201.159@tcp [14567.827156] Key type .llcrypt unregistered [14567.829359] Key type ._llcrypt unregistered