[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 767061886 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002391] x2apic enabled [ 0.003013] Switched APIC routing to physical x2apic. [ 0.004023] kvm-guest: setup PV IPIs [ 0.007803] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008034] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009020] pid_max: default: 32768 minimum: 301 [ 0.011080] LSM: Security Framework initializing [ 0.012083] Yama: becoming mindful. [ 0.013057] SELinux: Initializing. [ 0.014123] *** VALIDATE selinux *** [ 0.022701] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027217] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028189] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029137] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030154] *** VALIDATE tmpfs *** [ 0.033419] *** VALIDATE proc *** [ 0.034334] *** VALIDATE cgroup *** [ 0.035019] *** VALIDATE cgroup2 *** [ 0.037300] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038196] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039019] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040044] Spectre V2 : User space: Vulnerable [ 0.041017] Speculative Store Bypass: Vulnerable [ 0.045023] debug: unmapping init [mem 0xffffffffbbc59000-0xffffffffbbc60fff] [ 0.047427] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048945] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050038] ... version: 2 [ 0.051018] ... bit width: 48 [ 0.052015] ... generic registers: 4 [ 0.053021] ... value mask: 0000ffffffffffff [ 0.054035] ... max period: 00007fffffffffff [ 0.055020] ... fixed-purpose events: 3 [ 0.056021] ... event mask: 000000070000000f [ 0.057506] rcu: Hierarchical SRCU implementation. [ 0.060080] smp: Bringing up secondary CPUs ... [ 0.061971] x86: Booting SMP configuration: [ 0.062061] .... node #0, CPUs: #1 #2 #3 [ 0.073310] smp: Brought up 1 node, 4 CPUs [ 0.075022] smpboot: Max logical packages: 1 [ 0.076024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.121470] node 0 deferred pages initialised in 43ms [ 0.126256] devtmpfs: initialized [ 0.127363] x86/mm: Memory block size: 128MB [ 0.130327] gcov: version magic: 0x41383552 [ 0.132373] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.133192] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.134366] pinctrl core: initialized pinctrl subsystem [ 0.135350] [ 0.136011] ************************************************************* [ 0.137019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.138020] ** ** [ 0.139021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.140020] ** ** [ 0.141021] ** This means that this kernel is built to expose internal ** [ 0.142022] ** IOMMU data structures, which may compromise security on ** [ 0.143021] ** your system. ** [ 0.144018] ** ** [ 0.145081] ** If you see this message and you are not debugging the ** [ 0.146024] ** kernel, report this immediately to your vendor! ** [ 0.147025] ** ** [ 0.148069] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.149019] ************************************************************* [ 0.151070] NET: Registered protocol family 16 [ 0.152660] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.153148] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.154087] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.156153] cpuidle: using governor menu [ 0.160242] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.164975] PCI: Using configuration type 1 for base access [ 0.168288] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.181267] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.185077] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.187443] cryptd: max_cpu_qlen set to 1000 [ 0.189436] ACPI: Added _OSI(Module Device) [ 0.190000] ACPI: Added _OSI(Processor Device) [ 0.190016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.191000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.199146] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.210000] ACPI: Interpreter enabled [ 0.212130] ACPI: PM: (supports S0 S3 S4 S5) [ 0.214017] ACPI: Using IOAPIC for interrupt routing [ 0.219178] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.224569] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.232062] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.233055] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.236024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.240169] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.247384] acpiphp: Slot [2] registered [ 0.253062] acpiphp: Slot [5] registered [ 0.260156] acpiphp: Slot [6] registered [ 0.266182] acpiphp: Slot [7] registered [ 0.269150] acpiphp: Slot [8] registered [ 0.271164] acpiphp: Slot [9] registered [ 0.273132] acpiphp: Slot [10] registered [ 0.274130] acpiphp: Slot [3] registered [ 0.276144] acpiphp: Slot [4] registered [ 0.277333] acpiphp: Slot [11] registered [ 0.279099] acpiphp: Slot [12] registered [ 0.281219] acpiphp: Slot [13] registered [ 0.282117] acpiphp: Slot [14] registered [ 0.284166] acpiphp: Slot [15] registered [ 0.285000] acpiphp: Slot [16] registered [ 0.285000] acpiphp: Slot [17] registered [ 0.287130] acpiphp: Slot [18] registered [ 0.288239] acpiphp: Slot [19] registered [ 0.289000] acpiphp: Slot [20] registered [ 0.289000] acpiphp: Slot [21] registered [ 0.290101] acpiphp: Slot [22] registered [ 0.291126] acpiphp: Slot [23] registered [ 0.293138] acpiphp: Slot [24] registered [ 0.295136] acpiphp: Slot [25] registered [ 0.297123] acpiphp: Slot [26] registered [ 0.298176] acpiphp: Slot [27] registered [ 0.299105] acpiphp: Slot [28] registered [ 0.301130] acpiphp: Slot [29] registered [ 0.303128] acpiphp: Slot [30] registered [ 0.305131] acpiphp: Slot [31] registered [ 0.306098] PCI host bridge to bus 0000:00 [ 0.308026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.310045] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.312027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.314030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.316028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.317000] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.317000] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.321071] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.324314] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.351930] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.363000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.372035] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.376054] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.382020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.389000] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.397000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.404107] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.416101] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x1e0 took 19531 usecs [ 0.426119] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.440022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.458026] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.466023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.469000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.486020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.497024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.527026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.552221] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.563022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.572018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.598022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.620023] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.635021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.646021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.690020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.724375] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.746018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.759196] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.794284] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.813000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.822020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.833019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.866022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.889758] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.903080] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.915024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.964605] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.980084] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.985510] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.991007] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.000421] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.008296] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.025103] iommu: Default domain type: Passthrough [ 1.034081] SCSI subsystem initialized [ 1.043869] ACPI: bus type USB registered [ 1.052167] usbcore: registered new interface driver usbfs [ 1.062100] usbcore: registered new interface driver hub [ 1.063000] usbcore: registered new device driver usb [ 1.073220] pps_core: LinuxPPS API ver. 1 registered [ 1.078016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.086082] PTP clock support registered [ 1.098070] EDAC MC: Ver: 3.0.0 [ 1.101202] PCI: Using ACPI for IRQ routing [ 1.104509] NetLabel: Initializing [ 1.113016] NetLabel: domain hash size = 128 [ 1.117019] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.125075] NetLabel: unlabeled traffic allowed by default [ 1.130098] vgaarb: loaded [ 1.137360] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.141015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.163452] clocksource: Switched to clocksource kvm-clock [ 1.439417] VFS: Disk quotas dquot_6.6.0 [ 1.448170] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.458119] *** VALIDATE ramfs *** [ 1.461939] *** VALIDATE hugetlbfs *** [ 1.468289] pnp: PnP ACPI init [ 1.473740] pnp: PnP ACPI: found 6 devices [ 1.528792] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.536177] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.538331] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.540874] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.543447] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.546147] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.549327] NET: Registered protocol family 2 [ 1.552231] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.558065] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.561898] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.568148] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.571966] TCP: Hash tables configured (established 65536 bind 65536) [ 1.575586] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.579362] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.582684] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.586241] NET: Registered protocol family 1 [ 1.589053] RPC: Registered named UNIX socket transport module. [ 1.590907] RPC: Registered udp transport module. [ 1.592271] RPC: Registered tcp transport module. [ 1.594671] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.597855] NET: Registered protocol family 44 [ 1.601630] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.605893] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.610083] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.616467] PCI: CLS 0 bytes, default 64 [ 1.620935] Unpacking initramfs... [ 5.054872] debug: unmapping init [mem 0xffff91dc3cc54000-0xffff91dc3ffbffff] [ 5.065266] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 5.072236] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 5.082121] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 7.328255] Initialise system trusted keyrings [ 7.332416] Key type blacklist registered [ 7.336944] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 7.359098] zbud: loaded [ 7.364701] *** VALIDATE nfs *** [ 7.367079] *** VALIDATE nfs4 *** [ 7.370386] pstore: using deflate compression [ 7.400265] Platform Keyring initialized [ 7.801630] NET: Registered protocol family 38 [ 7.811446] Key type asymmetric registered [ 7.816951] Asymmetric key parser 'x509' registered [ 7.821612] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 7.831831] io scheduler mq-deadline registered [ 7.838687] io scheduler kyber registered [ 7.840663] io scheduler bfq registered [ 7.854392] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 7.864730] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 7.873569] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 7.881115] ACPI: Power Button [PWRF] [ 7.906693] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.932489] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.967759] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 7.985956] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 8.013427] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 8.068940] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 8.120382] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 8.126549] Non-volatile memory driver v1.3 [ 8.128230] Linux agpgart interface v0.103 [ 8.216783] virtio_blk virtio1: [vda] 149952 512-byte logical blocks (76.8 MB/73.2 MiB) [ 8.221937] vda: detected capacity change from 0 to 76775424 [ 8.245884] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 8.249884] vdb: detected capacity change from 0 to 1073741824 [ 8.289284] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.304235] vdc: detected capacity change from 0 to 2621440000 [ 8.341260] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.345289] vdd: detected capacity change from 0 to 2621440000 [ 8.378220] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.382980] vde: detected capacity change from 0 to 4294967296 [ 8.428281] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.456097] vdf: detected capacity change from 0 to 4294967296 [ 8.473127] libphy: Fixed MDIO Bus: probed [ 8.520937] usbcore: registered new interface driver usbserial_generic [ 8.533614] usbserial: USB Serial support registered for generic [ 8.540141] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 8.557781] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 8.559821] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 8.574315] mousedev: PS/2 mouse device common for all mice [ 8.582658] rtc_cmos 00:05: RTC can wake from S4 [ 8.598291] rtc_cmos 00:05: registered as rtc0 [ 8.606694] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 8.615876] intel_pstate: CPU model not supported [ 8.641709] hid: raw HID events driver (C) Jiri Kosina [ 8.654348] usbcore: registered new interface driver usbhid [ 8.655269] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 8.664967] usbhid: USB HID core driver [ 8.676567] drop_monitor: Initializing network drop monitor service [ 8.690705] Initializing XFRM netlink socket [ 8.713974] NET: Registered protocol family 10 [ 8.717804] Segment Routing with IPv6 [ 8.723305] NET: Registered protocol family 17 [ 8.728791] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 8.740608] mpls_gso: MPLS GSO support [ 8.768300] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 8.771443] RAS: Correctable Errors collector initialized. [ 8.791347] AVX version of gcm_enc/dec engaged. [ 8.797054] AES CTR mode by8 optimization enabled [ 9.026054] sched_clock: Marking stable (9026017778, 0)->(11515716605, -2489698827) [ 9.033573] registered taskstats version 1 [ 9.038712] Loading compiled-in X.509 certificates [ 9.046293] zswap: loaded using pool lzo/zbud [ 9.116417] Key type big_key registered [ 9.140465] Key type encrypted registered [ 9.142145] ima: No TPM chip found, activating TPM-bypass! [ 9.145880] ima: Allocated hash algorithm: sha1 [ 9.148639] ima: No architecture policies found [ 9.151421] evm: Initialising EVM extended attributes: [ 9.154137] evm: security.selinux [ 9.157516] evm: security.ima [ 9.159445] evm: security.capability [ 9.161212] evm: HMAC attrs: 0x1 [ 9.165278] rtc_cmos 00:05: setting system clock to 2026-09-10 16:44:25 UTC (1789058665) [ 9.178994] debug: unmapping init [mem 0xffffffffbcc03000-0xffffffffbcdfffff] [ 9.184801] debug: unmapping init [mem 0xffffffffbb982000-0xffffffffbbc58fff] [ 9.196118] Write protecting the kernel read-only data: 28672k [ 9.203517] debug: unmapping init [mem 0xffffffffba003000-0xffffffffba1fffff] [ 9.214083] debug: unmapping init [mem 0xffffffffba914000-0xffffffffba9fffff] [ 9.320454] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 9.335413] systemd[1]: Detected virtualization kvm. [ 9.338141] systemd[1]: Detected architecture x86-64. [ 9.341293] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 9.425522] systemd[1]: No hostname configured. [ 9.431108] systemd[1]: Set hostname to . [ 9.434293] random: systemd: uninitialized urandom read (16 bytes read) [ 9.443644] systemd[1]: Initializing machine ID from random generator. [ 9.779756] random: systemd: uninitialized urandom read (16 bytes read) [ 9.789415] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 9.827275] random: systemd: uninitialized urandom read (16 bytes read) [ 9.835872] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 9.851451] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Setup Virtual Console... Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 11.450588] device-mapper: uevent: version 1.0.3 [ 11.453557] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 13.145168] random: fast init done [ 13.293582] virtio_net virtio0 ens2: renamed from eth0 [ 13.534898] scsi host0: ata_piix [ 13.580842] scsi host1: ata_piix [ 13.584149] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 13.592144] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 18.681596] random: crng init done [ 18.682740] random: 7 urandom warning(s) missed due to ratelimiting [ 19.835857] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 22.127403] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.881712] printk: systemd: 26 output lines suppressed due to ratelimiting [ 25.676582] SELinux: Disabled at runtime. [ 25.801995] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.820531] systemd[1]: Detected virtualization kvm. [ 25.824918] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 27.436992] systemd[1]: initrd-switch-root.service: Succeeded. [ 27.449232] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 27.468186] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 27.475371] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 27.488388] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 27.542168] systemd[1]: Starting Journal Service... Starting Journal Service... [ 27.575520] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [[ 28.054026] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Created slice system-sshd\x2dkeygen.slice. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 29.182371] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 30.269641] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 30.327233] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 30.721150] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 30.745450] EDAC sbridge: Ver: 1.1.2 [ 34.787138] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 35.458465] NFS: Registering the id_resolver key type [ 35.461491] Key type id_resolver registered [ 35.464481] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg301-server login: [ 56.072106] hrtimer: interrupt took 16089203 ns [ 89.264564] libcfs: loading out-of-tree module taints kernel. [ 89.312727] Key type ._llcrypt registered [ 89.315432] Key type .llcrypt registered [ 89.401598] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 108.185366] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 110.455851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 110.483795] alg: No test for adler32 (adler32-zlib) [ 112.182467] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 113.284553] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 115.047228] Key type lgssc registered [ 117.701735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 140.217765] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 188.278670] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 200.573882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 200.626254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 201.816510] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 201.842876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 201.938317] Lustre: lustre-MDT0000: new disk, initializing [ 202.013751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.040763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 206.217502] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.124325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 219.221147] Lustre: 6526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 219.291395] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 219.300699] Lustre: Skipped 1 previous similar message [ 219.426120] Lustre: lustre-MDT0001: new disk, initializing [ 219.520893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 219.546730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 219.556461] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 225.217171] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.566819] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 240.104964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 240.364576] Lustre: lustre-OST0000: new disk, initializing [ 240.368528] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 240.374796] Lustre: 8466:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 240.460101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 245.815693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 245.824719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 245.900501] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 246.300334] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 259.476885] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 259.633618] Lustre: lustre-OST0001: new disk, initializing [ 259.641828] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 259.648697] Lustre: 9537:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 259.730936] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 265.502124] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 268.375473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 268.394482] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 268.440229] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 276.427888] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 281.750200] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 287.642744] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing check_logdir /tmp/testlogs/ [ 292.527261] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing yml_node [ 296.551180] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 298.543547] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 300.487079] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 301.858923] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 10 12:49:16 EDT 2026 [ 319.315757] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 321.976187] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 340.449273] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 340.452091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 340.464139] Lustre: Skipped 3 previous similar messages [ 340.478298] Lustre: Skipped 3 previous similar messages [ 345.571348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 345.585244] Lustre: Skipped 3 previous similar messages [ 346.591958] Lustre: server umount lustre-MDT0000 complete [ 350.358379] LustreError: 6518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789059006 with bad export cookie 18121598887312571866 [ 350.359191] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 350.369908] LustreError: 6518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 350.691204] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 350.694091] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 350.708169] Lustre: Skipped 1 previous similar message [ 355.808422] LustreError: 6532:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 355.808539] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 355.832420] LustreError: 6532:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 355.842260] Lustre: Skipped 2 previous similar messages [ 356.835562] Lustre: server umount lustre-MDT0001 complete [ 360.407809] Lustre: server umount lustre-OST0000 complete [ 364.039794] Lustre: server umount lustre-OST0001 complete [ 369.970908] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 377.877455] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 410.616032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 410.961898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 410.995633] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 411.078420] Lustre: lustre-MDT0000: new disk, initializing [ 411.235987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 411.264541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 416.531608] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 425.404614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 425.516403] Lustre: 15087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 425.587270] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 425.591221] Lustre: Skipped 1 previous similar message [ 425.697278] Lustre: lustre-MDT0001: new disk, initializing [ 425.825835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 425.894711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 425.908456] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 429.443436] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 439.283587] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 445.540664] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 453.484476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 453.814494] Lustre: lustre-OST0000: new disk, initializing [ 453.818960] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 453.826169] Lustre: 17057:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 453.925301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 455.679432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 455.696084] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 455.882047] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 460.578351] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 468.757483] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 472.783642] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 472.949850] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 476.601912] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 476.807598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 481.763088] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 481.769881] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 481.788597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 481.801344] Lustre: Skipped 1 previous similar message [ 484.683798] Lustre: server umount lustre-OST0000 complete [ 492.004112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 492.010173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 492.020043] Lustre: Skipped 2 previous similar messages [ 494.280323] Lustre: server umount lustre-MDT0000 complete [ 497.122100] LustreError: 15100:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 497.140510] LustreError: 15100:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 497.974589] LustreError: 15080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789059154 with bad export cookie 18121598887312573399 [ 497.983442] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 497.987171] LustreError: 15080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 498.363467] Lustre: server umount lustre-MDT0001 complete [ 503.396572] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 12:52:38 (1789059158) [ 521.970411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 522.538988] LustreError: 19934:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 522.617423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 527.756536] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 527.845948] LustreError: 19935:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.937560] LustreError: 19934:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.753914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 541.774428] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 549.335752] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 554.658621] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 556.938839] Lustre: Failing over lustre-MDT0000 [ 557.283285] Lustre: server umount lustre-MDT0000 complete [ 557.536717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 557.540850] LustreError: 19930:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.556476] Lustre: Skipped 1 previous similar message [ 557.593368] LustreError: 19930:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 572.768083] Lustre: 3658:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789059213/real 1789059213] req@ffff91dcbd804a80 x1875964087915008/t0(0) o400->MGC192.168.203.101@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789059229 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 572.790949] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 575.416355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 578.017489] LustreError: 21344:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.030324] LustreError: 21344:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 583.138870] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xfb7cda2cf861495d [ 583.155760] Lustre: MGC192.168.203.101@tcp: Connection restored to 0@lo (at 0@lo) [ 587.487633] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 588.772269] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 588.790759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 588.807645] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 596.575543] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 603.105373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 603.124861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 603.138218] Lustre: Skipped 2 previous similar messages [ 606.545376] Lustre: server umount lustre-MDT0000 complete [ 610.352671] LustreError: 19916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789059266 with bad export cookie 18121598887312574813 [ 610.367720] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 610.746650] Lustre: server umount lustre-MDT0001 complete [ 617.972990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 618.316333] LustreError: 23271:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 618.335451] LustreError: 23271:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 618.405875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 618.409608] Lustre: Skipped 2 previous similar messages [ 621.553512] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 628.325611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 633.369997] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 641.184299] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 648.402946] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 650.869889] Lustre: Failing over lustre-MDT0000 [ 651.080724] Lustre: server umount lustre-MDT0000 complete [ 652.969870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 652.980771] Lustre: Skipped 1 previous similar message [ 669.663835] Lustre: 3658:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789059309/real 1789059309] req@ffff91dc87f46680 x1875964087944448/t0(0) o400->MGC192.168.203.101@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789059325 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 669.693627] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 670.520111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 680.928728] LustreError: 3655:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff91db8312f800 x1875964087947648/t0(0) o250->MGC192.168.203.101@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 681.314159] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 685.951798] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 686.573984] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 686.584971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 686.597764] Lustre: Skipped 1 previous similar message [ 686.610076] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 696.050696] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 700.024710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 700.035032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 700.037796] Lustre: Skipped 2 previous similar messages [ 700.055911] Lustre: Skipped 1 previous similar message [ 706.409261] Lustre: server umount lustre-MDT0000 complete [ 710.427301] LustreError: 25936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789059366 with bad export cookie 18121598887312575758 [ 710.438548] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 710.449448] LustreError: 25936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 710.902749] Lustre: server umount lustre-MDT0001 complete [ 719.829689] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 12:56:13 (1789059373) [ 734.792290] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 743.930816] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 782.644401] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 792.072809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 792.296737] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 792.331084] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 792.454537] Lustre: lustre-MDT0000: new disk, initializing [ 792.568652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 792.585292] Lustre: Skipped 2 previous similar messages [ 792.616822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 798.744564] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 810.730426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 810.887348] Lustre: 29649:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 811.337414] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 815.804510] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 825.221766] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 831.073463] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 839.022972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 839.613119] Lustre: 31620:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 841.008179] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 841.142471] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 841.269912] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 841.311845] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 847.654284] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 857.162850] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 862.227718] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 882.146562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 882.150934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 882.150995] LustreError: Skipped 1 previous similar message [ 882.173905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 882.177078] Lustre: Skipped 3 previous similar messages [ 886.725471] Lustre: server umount lustre-OST0000 complete [ 896.250282] Lustre: server umount lustre-MDT0000 complete [ 897.505962] LustreError: 29659:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 897.520551] LustreError: 29659:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 21 previous similar messages [ 899.957191] LustreError: 29639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789059556 with bad export cookie 18121598887312576164 [ 899.958372] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 899.964637] LustreError: 29639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 900.416096] Lustre: server umount lustre-MDT0001 complete [ 911.193451] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 914.276218] Key type lgssc unregistered [ 914.572080] LNet: 33601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.588970] LNetError: 33601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.621884] LNet: Removed LNI 192.168.203.101@tcp [ 915.794800] Key type .llcrypt unregistered [ 915.800472] Key type ._llcrypt unregistered [ 939.950162] Key type ._llcrypt registered [ 939.954240] Key type .llcrypt registered [ 940.065314] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 941.284059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 941.422598] alg: No test for adler32 (adler32-zlib) [ 942.503196] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 942.787501] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 944.511213] Key type lgssc registered [ 945.931252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 954.084518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 954.130692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 955.629347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 960.383200] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 965.089856] LustreError: 35010:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 965.104801] LustreError: 35010:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 968.098054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 968.451905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 972.436276] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 979.383928] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 985.090313] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 990.753278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 991.086137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 993.150038] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 996.612591] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1000.342196] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1003.169937] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1015.926551] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 13:01:10 (1789059670) [ 1017.885848] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1019.795883] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1027.309872] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 13:01:21 (1789059681) [ 1035.132336] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 13:01:29 (1789059689) [ 1035.967179] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1044.415595] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1044.420889] Lustre: Skipped 2 previous similar messages [ 1046.376404] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 13:01:40 (1789059700) [ 1049.927123] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1049.938287] Lustre: Skipped 3 previous similar messages [ 1067.286826] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1067.295114] Lustre: Skipped 6 previous similar messages [ 1069.567116] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 13:02:04 (1789059724) [ 1072.090701] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1075.127387] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1082.291468] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 13:02:16 (1789059736) [ 1084.229358] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1084.233655] Lustre: Skipped 3 previous similar messages [ 1097.017814] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 13:02:31 (1789059751) [ 1113.058559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1113.061214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1113.063553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1113.081923] Lustre: Skipped 1 previous similar message [ 1117.560723] Lustre: server umount lustre-OST0000 complete [ 1122.272611] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1122.282243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1122.291178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1122.293440] Lustre: Skipped 1 previous similar message [ 1123.302153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1123.305425] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1127.286927] Lustre: server umount lustre-MDT0000 complete [ 1128.419734] LustreError: 35740:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1128.437509] LustreError: 35740:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1130.261771] LustreError: 36854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789059786 with bad export cookie 3433734660714018768 [ 1130.265442] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1130.273089] LustreError: 36854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1130.575835] Lustre: server umount lustre-MDT0001 complete [ 1140.726556] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1143.408733] Key type lgssc unregistered [ 1143.672837] LNet: 42868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1143.684341] LNetError: 42868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1143.700017] LNet: Removed LNI 192.168.203.101@tcp [ 1144.451639] Key type .llcrypt unregistered [ 1144.454448] Key type ._llcrypt unregistered [ 1160.820558] Key type ._llcrypt registered [ 1160.824204] Key type .llcrypt registered [ 1160.892980] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 1176.257525] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1177.518701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1177.543352] alg: No test for adler32 (adler32-zlib) [ 1178.666137] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 1178.893555] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 1180.544364] Key type lgssc registered [ 1181.827061] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1214.979845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1215.035997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1216.269519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1216.301915] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1216.396987] Lustre: lustre-MDT0000: new disk, initializing [ 1216.527757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1216.547406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1221.911359] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1231.520659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1231.636878] Lustre: 46019:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1231.708426] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1231.712357] Lustre: Skipped 1 previous similar message [ 1231.815420] Lustre: lustre-MDT0001: new disk, initializing [ 1231.905852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1231.940433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1231.949318] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1236.213972] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1244.889641] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1250.508124] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1257.015737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1257.290601] Lustre: lustre-OST0000: new disk, initializing [ 1257.293856] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1257.301307] Lustre: 47990:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1257.373887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1263.742342] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1264.770737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1264.786700] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1264.910033] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1273.253698] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1277.545183] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1277.810651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1282.270058] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1282.550966] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1285.605248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1285.605596] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1285.635869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1285.676039] Lustre: Skipped 1 previous similar message [ 1290.682059] Lustre: server umount lustre-OST0000 complete [ 1295.843132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1295.861028] Lustre: Skipped 1 previous similar message [ 1295.868794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1295.879261] Lustre: Skipped 1 previous similar message [ 1300.484162] Lustre: server umount lustre-MDT0000 complete [ 1300.959821] LustreError: 47521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1300.974164] LustreError: 47521:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1303.746446] LustreError: 46009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789059960 with bad export cookie 14341534920246572057 [ 1303.754948] LustreError: 46009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1303.761719] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1304.051170] Lustre: server umount lustre-MDT0001 complete [ 1312.495475] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 13:06:06 (1789059966) [ 1318.647138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1319.023228] LustreError: 50449:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1319.103915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1322.892252] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1324.520111] LustreError: 50450:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1329.633360] LustreError: 50449:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1330.592735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1330.924865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1335.213524] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1341.834470] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1346.628701] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1352.472606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1352.861809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1358.585149] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1364.299643] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1368.108536] LustreError: 52977:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1368.908895] Lustre: Modifying parameter general.timeout=1 in log params [ 1370.202496] Lustre: Modifying parameter general.timeout=3 in log params [ 1370.205845] Lustre: Skipped 1 previous similar message [ 1372.378803] Lustre: Modifying parameter general.timeout=6 in log params [ 1372.384958] Lustre: Skipped 2 previous similar messages [ 1376.665054] Lustre: Modifying parameter general.timeout=12 in log params [ 1376.669164] Lustre: Skipped 5 previous similar messages [ 1385.651836] Lustre: Modifying parameter general.timeout=22 in log params [ 1385.659406] Lustre: Skipped 9 previous similar messages [ 1402.220741] Lustre: Modifying parameter general.timeout=41 in log params [ 1402.225734] Lustre: Skipped 18 previous similar messages [ 1434.716899] Lustre: Modifying parameter general.timeout=83 in log params [ 1434.721609] Lustre: Skipped 41 previous similar messages [ 1483.529892] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 13:08:58 (1789060138) [ 1486.229743] Lustre: 59912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1487.750205] Lustre: 60011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1491.177758] Lustre: 60228:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1491.189287] Lustre: 60228:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1499.285255] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 in log params [ 1499.292577] Lustre: Skipped 57 previous similar messages [ 1517.004662] Lustre: 61781:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1517.011959] Lustre: 61781:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1517.024317] LustreError: 61781:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1520.019830] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 13:09:34 (1789060174) [ 1529.824478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1529.826345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1529.839920] Lustre: Skipped 3 previous similar messages [ 1529.852207] Lustre: Skipped 2 previous similar messages [ 1534.948295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1535.887253] Lustre: server umount lustre-MDT0000 complete [ 1538.564849] LustreError: 53966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789060194 with bad export cookie 14341534920246572939 [ 1538.568273] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1538.581875] LustreError: 53966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1538.850824] Lustre: server umount lustre-MDT0001 complete [ 1552.800029] Lustre: server umount lustre-OST0000 complete [ 1579.236298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1579.462594] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1579.741422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1584.326842] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1594.625710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1594.841207] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1594.894445] Lustre: 64311:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1600.197091] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1609.030178] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1613.681547] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1620.038498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1620.287413] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1620.576788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1620.580977] Lustre: Skipped 1 previous similar message [ 1626.148583] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1633.656311] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1635.347434] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1635.357284] Lustre: Skipped 12 previous similar messages [ 1635.410266] Lustre: 66881:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1646.054430] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1646.066583] LustreError: Skipped 1 previous similar message [ 1646.070170] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1646.092087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1646.097444] Lustre: Skipped 2 previous similar messages [ 1648.747889] Lustre: server umount lustre-OST0000 complete [ 1656.801867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1656.804301] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1656.813194] LustreError: Skipped 1 previous similar message [ 1656.815063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1656.815070] Lustre: Skipped 1 previous similar message [ 1656.851574] Lustre: Skipped 2 previous similar messages [ 1658.702145] Lustre: server umount lustre-MDT0000 complete [ 1661.925359] LustreError: 65227:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1661.951358] LustreError: 65227:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1661.971523] LustreError: 64303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789060318 with bad export cookie 14341534920246575900 [ 1661.978033] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1661.983606] LustreError: 64303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1662.298749] Lustre: server umount lustre-MDT0001 complete [ 1672.585215] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1675.256732] Key type lgssc unregistered [ 1675.579450] LNet: 68088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1675.593315] LNetError: 68088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1675.615357] LNet: Removed LNI 192.168.203.101@tcp [ 1676.477132] Key type .llcrypt unregistered [ 1676.482299] Key type ._llcrypt unregistered [ 1699.236428] Key type ._llcrypt registered [ 1699.241187] Key type .llcrypt registered [ 1699.375488] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1700.368338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1700.417085] alg: No test for adler32 (adler32-zlib) [ 1701.531910] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 1701.809398] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 1703.511310] Key type lgssc registered [ 1704.682561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1712.225346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1712.249612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1713.673631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1717.703606] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1722.849037] LustreError: 69498:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1722.865915] LustreError: 69498:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1724.737631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1725.318324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1728.951816] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1734.778619] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1739.803942] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1745.300795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1745.791986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1750.777521] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1757.290728] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1764.589197] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1771.491490] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1771.501985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1771.519375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1773.538775] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1773.540303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1773.557524] Lustre: Skipped 1 previous similar message [ 1773.561838] Lustre: Skipped 1 previous similar message [ 1776.957610] Lustre: server umount lustre-MDT0000 complete [ 1778.666838] LustreError: 69497:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1778.696787] LustreError: 69497:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1780.546916] LustreError: 69478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789060436 with bad export cookie 13226443606776669265 [ 1780.554161] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1780.555811] LustreError: 69478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1780.866434] Lustre: server umount lustre-MDT0001 complete [ 1794.044605] Lustre: server umount lustre-OST0000 complete [ 1819.822157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1820.257758] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1820.317852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1820.736311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1825.091565] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1832.727899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1832.840290] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1832.887052] Lustre: 74367:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1832.979073] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1832.982556] Lustre: Skipped 1 previous similar message [ 1833.283015] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1837.349747] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1845.129834] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1851.378566] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1857.619764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1857.793631] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1858.280784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1864.744655] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1871.559421] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1873.291764] Lustre: 76940:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1875.177885] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1882.592586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1882.604129] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.630664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1886.410367] Lustre: server umount lustre-OST0000 complete [ 1893.345677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1893.361689] Lustre: Skipped 1 previous similar message [ 1893.369396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1893.378233] Lustre: Skipped 3 previous similar messages [ 1896.293355] Lustre: server umount lustre-MDT0000 complete [ 1898.464246] LustreError: 74379:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1898.488609] LustreError: 74379:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1899.943507] LustreError: 74359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789060556 with bad export cookie 13226443606776670105 [ 1899.946484] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1899.963325] LustreError: 74359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1900.350755] Lustre: server umount lustre-MDT0001 complete [ 1911.082528] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1913.744369] Key type lgssc unregistered [ 1914.134449] LNet: 78152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1914.144312] LNetError: 78152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1914.169502] LNet: Removed LNI 192.168.203.101@tcp [ 1915.050760] Key type .llcrypt unregistered [ 1915.053190] Key type ._llcrypt unregistered [ 1938.712881] Key type ._llcrypt registered [ 1938.719128] Key type .llcrypt registered [ 1938.854164] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1940.064103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1940.206125] alg: No test for adler32 (adler32-zlib) [ 1941.305785] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 1941.524145] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 1943.160305] Key type lgssc registered [ 1944.383582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1952.137141] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1952.154830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1953.687082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1957.860891] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1962.978177] LustreError: 79562:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1962.987053] LustreError: 79562:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1965.245135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1965.694903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1969.902238] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1977.091736] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1981.928505] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1988.500692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1989.169624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1996.199970] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2003.221466] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2008.690264] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2009.532490] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2011.528317] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2011.540472] Lustre: Skipped 1 previous similar message [ 2013.985685] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2013.994217] Lustre: Skipped 2 previous similar messages [ 2018.363802] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2018.375457] Lustre: Skipped 4 previous similar messages [ 2026.665392] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2026.682481] Lustre: Skipped 9 previous similar messages [ 2042.853662] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2042.859589] Lustre: Skipped 19 previous similar messages [ 2075.045767] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2075.050208] Lustre: Skipped 37 previous similar messages [ 2102.644317] Lustre: 87661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2105.231412] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 13:19:19 (1789060759) [ 2106.972578] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2108.751804] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 13:19:23 (1789060763) [ 2116.065585] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2116.078632] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2116.085119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2116.096888] Lustre: Skipped 1 previous similar message [ 2116.122356] Lustre: Skipped 1 previous similar message [ 2121.184742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2121.194379] Lustre: Skipped 1 previous similar message [ 2126.312763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2126.318692] Lustre: Skipped 1 previous similar message [ 2127.327243] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2127.449356] Lustre: server umount lustre-OST0000 complete [ 2131.432152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2131.458151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2136.547360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2136.567169] Lustre: Skipped 2 previous similar messages [ 2145.247116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2145.408360] Lustre: server umount lustre-MDT0000 complete [ 2146.792390] LustreError: 80292:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2146.814722] LustreError: 80292:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2148.825822] LustreError: 79540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789060805 with bad export cookie 4558188408339809427 [ 2148.827450] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2148.849630] LustreError: 79540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2149.201638] Lustre: server umount lustre-MDT0001 complete [ 2159.582763] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 2162.440588] Key type lgssc unregistered [ 2162.713298] LNet: 88913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2162.720397] LNetError: 88913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2162.740734] LNet: Removed LNI 192.168.203.101@tcp [ 2163.589765] Key type .llcrypt unregistered [ 2163.594534] Key type ._llcrypt unregistered [ 2167.650286] Key type ._llcrypt registered [ 2167.656784] Key type .llcrypt registered [ 2167.795366] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2172.479053] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules [ 2173.264419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2173.286288] alg: No test for adler32 (adler32-zlib) [ 2174.429930] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 2174.731417] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 2176.441523] Key type lgssc registered [ 2177.543327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2178.748891] LustreError: 90110:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2193.813376] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 2201.008817] LustreError: 90110:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2202.418545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2202.443235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2204.140369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2209.115055] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2214.408121] LustreError: 91134:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2214.434326] LustreError: 91134:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2218.634055] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 13:21:12 (1789060872) [ 2219.494717] LustreError: 91135:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2223.724102] LustreError: 91166:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2223.745293] LustreError: 91166:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2223.761491] LustreError: 91166:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2224.327352] Lustre: server umount lustre-MDT0000 complete [ 2237.670889] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 2240.367644] Key type lgssc unregistered [ 2240.779934] LNet: 92660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2240.797467] LNetError: 92660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2240.820198] LNet: Removed LNI 192.168.203.101@tcp [ 2241.618997] Key type .llcrypt unregistered [ 2241.622222] Key type ._llcrypt unregistered [ 2258.512408] Key type ._llcrypt registered [ 2258.514746] Key type .llcrypt registered [ 2258.630292] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 2259.764744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2259.798182] alg: No test for adler32 (adler32-zlib) [ 2260.837948] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 2261.079386] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 2262.727202] Key type lgssc registered [ 2263.709490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2269.808085] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2269.826837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2271.393773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2275.572589] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2281.011682] LustreError: 93877:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2281.032658] LustreError: 93877:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2285.237155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2286.032534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2291.641164] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2299.212653] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2304.626713] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2310.676247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2311.195857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2317.813667] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2325.816443] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2372.080193] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 13:23:46 (1789061026) [ 2374.064428] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2375.761028] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 13:23:50 (1789061030) [ 2381.796363] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2381.798744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2381.818512] Lustre: Skipped 1 previous similar message [ 2381.833890] Lustre: Skipped 2 previous similar messages [ 2386.291069] Lustre: server umount lustre-MDT0000 complete [ 2386.913897] LustreError: 94608:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.948173] LustreError: 94608:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2389.482232] LustreError: 94618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789061045 with bad export cookie 14477272927806227201 [ 2389.483704] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2389.503597] LustreError: 94618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2389.946911] Lustre: server umount lustre-MDT0001 complete [ 2404.038201] Lustre: server umount lustre-OST0000 complete [ 2412.800330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2413.170604] LustreError: 98154:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2413.332743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2417.494890] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2418.663356] LustreError: 98155:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2422.760124] LustreError: 98154:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2425.360212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2426.169813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2430.720866] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2438.253188] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2444.336173] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2457.487940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2457.759966] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2457.772515] LustreError: 98143:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2457.793804] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2457.800467] LustreError: 100247:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2457.810759] LustreError: 100247:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2457.819273] LustreError: 100247:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2457.906511] Lustre: server umount lustre-OST0000 complete [ 2457.911356] LustreError: 100247:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2466.328192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2466.498072] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2466.503396] LustreError: 98143:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2466.530702] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2466.539752] LustreError: 100714:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2466.548925] LustreError: 100714:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2466.555219] LustreError: 100714:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2466.622419] Lustre: server umount lustre-OST0000 complete [ 2466.625855] LustreError: 100714:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2476.116278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2476.318227] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2476.333608] Lustre: Found index 0 for lustre-OST0000, updating log [ 2476.347328] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2476.446120] Lustre: lustre-OST0000: new disk, initializing [ 2476.450668] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2476.461412] Lustre: 101250:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2476.850188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2482.946278] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2486.369466] Lustre: Failing over lustre-OST0000 [ 2486.561655] Lustre: server umount lustre-OST0000 complete [ 2492.387873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2492.402772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2492.418404] Lustre: Skipped 1 previous similar message [ 2492.421204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2493.411435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2493.432391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2493.437271] Lustre: Skipped 1 previous similar message [ 2496.548846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2496.556208] Lustre: Skipped 1 previous similar message [ 2500.748125] Lustre: server umount lustre-MDT0000 complete [ 2501.601915] LustreError: 98154:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2501.618483] LustreError: 98154:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2504.259972] LustreError: 101249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789061160 with bad export cookie 14477272927806229945 [ 2504.262350] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2504.269494] LustreError: 101249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2504.584264] Lustre: server umount lustre-MDT0001 complete [ 2513.571642] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 13:26:07 (1789061167) [ 2529.702663] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 2539.608820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2540.151817] LustreError: 103594:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2540.314035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2544.611973] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2553.363230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2558.532811] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2561.315345] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2567.647793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2568.540736] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2574.512486] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2578.791075] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2578.803494] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2578.814955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2583.135140] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2583.256464] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2583.364477] Lustre: lustre-OST0001: new disk, initializing [ 2583.376253] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2583.386476] Lustre: 105913:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2583.829803] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2583.861570] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2583.875551] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2583.883666] Lustre: Skipped 2 previous similar messages [ 2584.877829] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:03 [ 2584.915601] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2584.916201] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2584.944309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2590.359690] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2592.837603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2592.852658] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2592.955536] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2598.146760] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2608.066323] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2613.738372] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2613.741406] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2613.754800] Lustre: Skipped 2 previous similar messages [ 2613.783294] Lustre: Skipped 2 previous similar messages [ 2617.308610] Lustre: server umount lustre-MDT0001 complete [ 2618.874054] LustreError: 103594:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2618.891278] LustreError: 103594:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2627.705271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2627.859692] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2627.867606] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2627.961542] Lustre: 103582:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2629.884598] LustreError: 107494:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2629.912181] LustreError: 107494:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2629.938568] LustreError: 103586:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2630.037776] LustreError: 107501:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.101@tcp: cfg command failed: rc = -17 [ 2630.048935] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2630.048935] [ 2633.726193] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2634.688280] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2643.473886] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 13:28:18 (1789061298) [ 2659.841973] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 2677.663135] Lustre: 109530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2702.345028] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2706.418592] Lustre: 110666:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2791.906033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2791.915439] LustreError: Skipped 1 previous similar message [ 2791.920499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2791.944634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2791.964323] Lustre: Skipped 1 previous similar message [ 2797.089333] Lustre: server umount lustre-MDT0000 complete [ 2797.546894] LustreError: 106713:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2797.573778] LustreError: 106713:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 2801.247398] LustreError: 103574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789061457 with bad export cookie 14477272927806230946 [ 2801.252026] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2801.257912] LustreError: 103574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2801.630552] Lustre: server umount lustre-MDT0001 complete [ 2816.126117] Lustre: server umount lustre-OST0000 complete [ 2818.015965] Lustre: 93187:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789061458/real 1789061458] req@ffff91dc8854b100 x1875966340456832/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789061474 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2818.051289] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2818.066154] Lustre: Skipped 3 previous similar messages [ 2819.627561] Lustre: server umount lustre-OST0001 complete [ 2824.843796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2834.357921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2843.051638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2853.116329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2873.298671] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 2883.502837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2883.535992] Lustre: lustre-MDT0000: reset Object Index mappings [ 2884.093889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2884.097982] Lustre: Skipped 1 previous similar message [ 2888.532584] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2896.451932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2896.467313] Lustre: lustre-MDT0001: reset Object Index mappings [ 2902.015334] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2905.055605] Lustre: 115766:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2911.611492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2918.522583] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2921.787964] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 2927.079142] LustreError: 116118:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2927.104863] LustreError: 116118:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2927.598192] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2930.417784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 2930.431613] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 2930.435536] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 2934.666973] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2942.195960] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2945.756522] Lustre: 117641:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2999.232537] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 13:34:13 (1789061653) [ 3027.941628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3027.951739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3027.959085] Lustre: Skipped 2 previous similar messages [ 3027.970913] Lustre: Skipped 5 previous similar messages [ 3031.938073] Lustre: server umount lustre-MDT0000 complete [ 3035.874936] LustreError: 114598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789061692 with bad export cookie 14477272927806278063 [ 3035.882166] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3035.887926] LustreError: 114598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3042.460966] Lustre: server umount lustre-MDT0001 complete [ 3052.573802] Lustre: server umount lustre-OST0000 complete [ 3062.508945] Lustre: server umount lustre-OST0001 complete [ 3070.921640] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 3083.844462] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 3118.630320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3118.855906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3118.888547] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3119.016421] Lustre: lustre-MDT0000: new disk, initializing [ 3119.092834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3119.102692] Lustre: Skipped 3 previous similar messages [ 3119.119625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3123.314861] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3131.558967] Lustre: server umount lustre-MDT0000 complete [ 3140.914510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3141.209472] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3141.225561] Lustre: 121913:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3141.257433] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3146.176586] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3150.632157] Lustre: server umount lustre-MDT0000 complete [ 3158.106548] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 13:36:52 (1789061812) [ 3159.654714] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3161.506435] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 13:36:55 (1789061815) [ 3182.893279] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 3192.712037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3197.059226] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3205.002641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3205.072405] Lustre: 123963:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3205.117761] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3205.122412] Lustre: Skipped 1 previous similar message [ 3205.171550] Lustre: lustre-MDT0001: new disk, initializing [ 3205.277816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3205.287686] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3209.561886] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3214.364932] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3220.945395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3221.214618] Lustre: lustre-OST0000: new disk, initializing [ 3221.220303] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3221.225674] Lustre: 125540:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3223.081628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3223.095256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3223.201103] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3227.302950] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3237.567400] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3237.693796] Lustre: lustre-OST0001: new disk, initializing [ 3237.697395] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3237.702738] Lustre: 126409:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3239.505611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3239.514647] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3239.696565] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3243.592644] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3252.408190] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3255.563989] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3631.253522] Lustre: 132819:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3631.261954] Lustre: 132819:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3631.280166] Lustre: 132819:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3631.299022] Lustre: 132819:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3631.310270] Lustre: 132819:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3631.321040] Lustre: 132819:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3969.114925] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:50:21 (1789062621) [ 3987.428305] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3987.433851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3987.447827] Lustre: Skipped 6 previous similar messages [ 3987.456877] Lustre: Skipped 7 previous similar messages [ 3990.821255] Lustre: server umount lustre-MDT0000 complete [ 3992.556590] LustreError: 123970:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3992.583738] LustreError: 123970:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 3995.172554] LustreError: 123954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789062651 with bad export cookie 14477272927806346509 [ 3995.181332] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3995.184300] LustreError: 123954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3997.665090] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3997.672478] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3997.690762] Lustre: Skipped 1 previous similar message [ 3997.705115] Lustre: Skipped 1 previous similar message [ 4001.885190] Lustre: server umount lustre-MDT0001 complete [ 4012.371955] Lustre: server umount lustre-OST0000 complete [ 4022.742979] Lustre: server umount lustre-OST0001 complete [ 4030.923188] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 4042.940613] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 4083.534347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4083.790344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4083.834174] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4083.971657] Lustre: lustre-MDT0000: new disk, initializing [ 4084.108858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4084.115463] Lustre: Skipped 5 previous similar messages [ 4084.152025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4090.270541] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4102.691592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4102.827264] Lustre: 135883:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4102.920695] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4102.928548] Lustre: Skipped 1 previous similar message [ 4103.056521] Lustre: lustre-MDT0001: new disk, initializing [ 4103.248177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4103.260461] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4107.702317] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4118.359368] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4125.059480] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4133.086283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4133.450978] Lustre: lustre-OST0000: new disk, initializing [ 4133.454643] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4133.467741] Lustre: 137850:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4133.629537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4133.638757] Lustre: Skipped 1 previous similar message [ 4134.768086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4134.791719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4134.937071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4141.364174] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4151.810436] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4163.001786] Lustre: lustre-MDD0000: changelog on [ 4166.076426] Lustre: lustre-MDD0001: changelog on [ 5960.370780] Lustre: lustre-MDD0001: changelog off [ 5962.467894] Lustre: 138336:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5963.463447] Lustre: lustre-MDD0000: changelog off [ 5970.919566] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5970.934296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5970.947129] Lustre: Skipped 1 previous similar message [ 5970.966575] Lustre: Skipped 4 previous similar messages [ 5976.040236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5976.047529] Lustre: Skipped 1 previous similar message [ 5976.908727] Lustre: server umount lustre-OST0000 complete [ 5981.164724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5981.174513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5981.184136] Lustre: Skipped 1 previous similar message [ 5986.284748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5986.298170] Lustre: Skipped 2 previous similar messages [ 5987.405236] Lustre: server umount lustre-MDT0000 complete [ 5991.036225] LustreError: 135876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789064647 with bad export cookie 14477272927807166720 [ 5991.038796] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5991.057407] LustreError: 135876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5991.513726] Lustre: server umount lustre-MDT0001 complete [ 6003.264263] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 6006.319480] Key type lgssc unregistered [ 6006.637361] LNet: 146999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6006.642222] LNetError: 146999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6006.669298] LNet: Removed LNI 192.168.203.101@tcp [ 6007.776282] Key type .llcrypt unregistered [ 6007.780498] Key type ._llcrypt unregistered [ 6028.643350] Key type ._llcrypt registered [ 6028.646730] Key type .llcrypt registered [ 6028.791785] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 6045.041759] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 6046.019201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6046.127825] alg: No test for adler32 (adler32-zlib) [ 6047.201599] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 6047.442712] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 6049.111222] Key type lgssc registered [ 6050.406679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6089.188359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6089.261466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6090.728762] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6090.772061] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6090.980566] Lustre: lustre-MDT0000: new disk, initializing [ 6091.172992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6091.208584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6096.661410] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6107.923510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6108.089737] Lustre: 150199:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6108.175859] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6108.184678] Lustre: Skipped 1 previous similar message [ 6108.278910] Lustre: lustre-MDT0001: new disk, initializing [ 6108.398702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6108.466315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6108.497263] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6113.044639] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6122.585309] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6129.030033] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6135.823306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6136.225443] Lustre: lustre-OST0000: new disk, initializing [ 6136.229403] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6136.237090] Lustre: 152166:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6136.331880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6143.200893] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6145.620546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6145.649933] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6145.797532] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6153.173502] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6160.401897] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_module obdecho/obdecho [ 6162.476883] LustreError: 153172:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6162.484639] LustreError: 153172:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6164.331656] LustreError: 153268:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6175.203194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6175.215256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6175.233837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6176.743538] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6176.744266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6176.772950] Lustre: Skipped 2 previous similar messages [ 6181.020574] Lustre: server umount lustre-MDT0000 complete [ 6185.459832] LustreError: 150191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789064841 with bad export cookie 10409842132182581925 [ 6185.460843] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6185.481491] LustreError: 150191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6186.126931] Lustre: server umount lustre-MDT0001 complete [ 6201.325958] Lustre: server umount lustre-OST0000 complete [ 6222.843361] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 6235.297985] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 6272.054336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6272.375529] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6272.411611] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6272.525615] Lustre: lustre-MDT0000: new disk, initializing [ 6272.634572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6272.658878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6277.348520] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6288.425764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6288.568745] Lustre: 156985:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6288.660536] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6288.669787] Lustre: Skipped 1 previous similar message [ 6288.813389] Lustre: lustre-MDT0001: new disk, initializing [ 6288.942858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6289.013378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6289.024639] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6293.876728] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6304.039740] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6309.913719] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6316.538218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6316.721286] Lustre: lustre-OST0000: new disk, initializing [ 6316.724333] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6316.728844] Lustre: 158955:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6316.793817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6321.935758] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6322.221277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6322.236631] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6322.387811] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6330.879650] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6336.162346] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6336.419705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6341.615962] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6341.987578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6348.264473] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6348.271292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6348.281923] LustreError: Skipped 1 previous similar message [ 6348.301092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6348.306987] Lustre: Skipped 2 previous similar messages [ 6348.328742] Lustre: Skipped 1 previous similar message [ 6350.820341] Lustre: server umount lustre-OST0000 complete [ 6355.426026] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6355.442145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6355.458917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6361.021217] Lustre: server umount lustre-MDT0000 complete [ 6363.624435] LustreError: 156998:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6363.644958] LustreError: 156998:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6364.450451] LustreError: 156977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789065020 with bad export cookie 10409842132182582884 [ 6364.461550] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6364.467130] LustreError: 156977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6364.869049] Lustre: server umount lustre-MDT0001 complete [ 6368.540266] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 14:30:22 (1789065022) [ 6370.527373] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6372.403174] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 14:30:26 (1789065026) [ 6380.750135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6381.356346] LustreError: 161368:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6381.476647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6385.908442] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6386.660764] LustreError: 161369:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6390.755472] LustreError: 161368:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6393.920575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6398.839667] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6406.056989] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6411.783821] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6419.529921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6419.941130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6419.954533] Lustre: Skipped 1 previous similar message [ 6427.248384] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6435.892610] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6447.766695] Lustre: server umount lustre-MDT0000 complete [ 6449.644797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6449.647734] LustreError: 161365:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6449.662967] Lustre: Skipped 3 previous similar messages [ 6449.677446] LustreError: 161365:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6451.329239] LustreError: 162105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789065107 with bad export cookie 10409842132182583766 [ 6451.329692] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6451.340231] LustreError: 162105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6452.309241] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.1@tcp (stopping) [ 6452.322903] Lustre: Skipped 2 previous similar messages [ 6457.752935] Lustre: server umount lustre-MDT0001 complete [ 6464.708247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6479.935387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6480.261001] LustreError: 165208:0:(llog.c:1655:llog_backup()) MGC192.168.203.101@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6480.268857] Lustre: 165208:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.101@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6481.376314] LustreError: 165213:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6481.412887] LustreError: 165213:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6498.311161] LustreError: 165215:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6498.330025] LustreError: 165215:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [ 6532.577868] LustreError: 163203:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6532.601825] LustreError: 163203:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 26 previous similar messages [ 6541.282236] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6541.286140] Lustre: 165208:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.101@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6582.263342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6587.290364] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6587.374496] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6596.110218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6597.343154] LustreError: 165951:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 6597.359904] LustreError: 165951:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000ffcb0aa0 x1875970310137856/t0(0) o250->MGC192.168.203.101@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1789065253 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6597.395323] LustreError: 165951:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 6597.607448] LustreError: 163203:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6597.631702] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x90773043f8a3630c [ 6597.645413] LustreError: 163203:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 42 previous similar messages [ 6597.675833] Lustre: MGC192.168.203.101@tcp: Connection restored to 0@lo (at 0@lo) [ 6602.135173] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6603.252925] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6610.341672] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6612.328954] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6620.376896] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6622.310274] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6644.193339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6644.198860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6644.206543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6644.205860] Lustre: Skipped 2 previous similar messages [ 6644.215618] Lustre: Skipped 3 previous similar messages [ 6646.399105] Lustre: server umount lustre-MDT0000 complete [ 6650.000563] LustreError: 161348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789065306 with bad export cookie 10409842132182590220 [ 6650.005900] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6650.015486] LustreError: 161348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6650.567683] Lustre: server umount lustre-MDT0001 complete [ 6664.536559] Lustre: server umount lustre-OST0000 complete [ 6682.107969] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 6691.775974] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 6725.602375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6725.786643] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6725.835140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6725.881549] Lustre: lustre-MDT0000: new disk, initializing [ 6725.960568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6725.968256] Lustre: Skipped 1 previous similar message [ 6726.000164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6731.009771] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6742.901808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6743.112941] Lustre: 170310:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6743.524117] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6748.085709] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6756.871274] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6763.061784] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6769.035117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6769.365788] Lustre: lustre-OST0000: new disk, initializing [ 6769.371178] Lustre: Skipped 1 previous similar message [ 6769.374629] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6769.378117] Lustre: Skipped 2 previous similar messages [ 6769.384212] Lustre: 172281:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6774.876355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6774.895916] Lustre: Skipped 1 previous similar message [ 6774.920913] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6775.020919] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6776.302786] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6787.673761] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6792.451497] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6792.594330] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6796.398686] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6796.621139] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6800.866833] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6800.868051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6800.875595] LustreError: Skipped 1 previous similar message [ 6800.885801] Lustre: Skipped 1 previous similar message [ 6800.888531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6800.895649] Lustre: Skipped 1 previous similar message [ 6805.003416] Lustre: server umount lustre-OST0000 complete [ 6810.080511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6815.928368] Lustre: server umount lustre-MDT0000 complete [ 6816.236457] LustreError: 170318:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6816.265216] LustreError: 170318:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6820.263062] LustreError: 172280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789065476 with bad export cookie 10409842132182607041 [ 6820.263869] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6820.281384] LustreError: 172280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6820.596604] Lustre: server umount lustre-MDT0001 complete [ 6830.008263] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 14:38:04 (1789065484) [ 6836.548946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6842.108285] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6850.711859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6855.813156] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6862.478672] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6867.666607] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6874.278933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6881.172659] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6888.879321] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6898.906287] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 14:39:13 (1789065553) [ 6907.364887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6907.370570] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6907.376023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6907.376031] Lustre: Skipped 3 previous similar messages [ 6907.384498] Lustre: Skipped 3 previous similar messages [ 6907.419698] LustreError: Skipped 1 previous similar message [ 6909.470442] Lustre: server umount lustre-OST0000 complete [ 6919.478711] Lustre: server umount lustre-MDT0000 complete [ 6923.226082] LustreError: 174672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789065579 with bad export cookie 10409842132182607930 [ 6923.230605] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6923.238251] LustreError: 174672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6923.574413] Lustre: server umount lustre-MDT0001 complete [ 6934.749306] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 6937.775244] Key type lgssc unregistered [ 6938.050872] LNet: 178419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6938.059394] LNetError: 178419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6938.087677] LNet: Removed LNI 192.168.203.101@tcp [ 6938.994171] Key type .llcrypt unregistered [ 6938.996205] Key type ._llcrypt unregistered [ 6957.565390] Key type ._llcrypt registered [ 6957.567615] Key type .llcrypt registered [ 6957.676031] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 6958.724492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6958.749366] alg: No test for adler32 (adler32-zlib) [ 6959.803363] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 6960.016964] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 6961.663197] Key type lgssc registered [ 6962.773848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6969.566774] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6969.604687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6985.185301] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6990.303413] LustreError: 179662:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.101@tcp: failed processing log, type 4: rc = -110 [ 6995.424085] LustreError: 179662:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6995.451387] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6995.466588] LustreError: MGC192.168.203.101@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6995.491816] LustreError: 179662:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6995.508020] LustreError: 179662:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 6995.517375] LustreError: 179662:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 6995.582908] Lustre: server umount lustre-OST0000 complete [ 6995.588898] LustreError: 179662:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7003.740302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7005.266972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7009.820324] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7015.398139] LustreError: 180162:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7015.422787] LustreError: 180162:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7017.330864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7017.637512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7021.503604] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7030.210300] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7036.461931] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7044.632510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7044.814965] LustreError: 181977:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7044.828210] Lustre: 181977:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.101@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7045.005644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7050.333159] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7056.920323] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7058.912509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7058.918151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7058.932696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7060.457653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7060.458138] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7060.494017] Lustre: Skipped 1 previous similar message [ 7062.561669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7062.566229] Lustre: Skipped 4 previous similar messages [ 7064.991829] Lustre: server umount lustre-MDT0000 complete [ 7067.617093] LustreError: 181225:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7067.627673] LustreError: 181225:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7068.565384] LustreError: 180143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789065724 with bad export cookie 14731526515077823716 [ 7068.566784] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7068.579957] LustreError: 180143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7068.703253] LustreError: 180890:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000533d17bb x1875971267391232/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7068.947524] Lustre: server umount lustre-MDT0001 complete [ 7082.222715] Lustre: server umount lustre-OST0000 complete [ 7092.658207] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 7103.071507] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 7140.495510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7140.766117] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7140.807962] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7140.922136] Lustre: lustre-MDT0000: new disk, initializing [ 7141.020234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7141.037974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7146.037743] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7157.124473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7157.274895] Lustre: 185501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7157.327167] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7157.330696] Lustre: Skipped 1 previous similar message [ 7157.473745] Lustre: lustre-MDT0001: new disk, initializing [ 7157.619556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7157.659838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7157.669084] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7162.369784] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7172.328982] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7178.748765] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7185.859702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7186.297617] Lustre: lustre-OST0000: new disk, initializing [ 7186.305611] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7186.318907] Lustre: 187468:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7186.469400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7192.205506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7192.234375] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7192.384526] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7192.396985] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7203.340676] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7209.903995] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7210.319955] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7217.331609] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7217.810714] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7223.273268] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7223.278878] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7223.309068] Lustre: Skipped 1 previous similar message [ 7223.319872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7223.338178] Lustre: Skipped 1 previous similar message [ 7226.870439] Lustre: server umount lustre-OST0000 complete [ 7233.520155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7233.548893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7233.563486] Lustre: Skipped 1 previous similar message [ 7237.542556] Lustre: server umount lustre-MDT0000 complete [ 7238.630757] LustreError: 185512:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7241.236797] LustreError: 186295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789065897 with bad export cookie 14731526515077824437 [ 7241.241437] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7241.250470] LustreError: 186295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7241.536170] Lustre: server umount lustre-MDT0001 complete [ 7251.054877] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 14:45:05 (1789065905) [ 7270.214289] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 7273.068628] Key type lgssc unregistered [ 7273.277537] LNet: 190412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7273.292814] LNetError: 190412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7273.302663] LNet: Removed LNI 192.168.203.101@tcp [ 7274.137903] Key type .llcrypt unregistered [ 7274.140422] Key type ._llcrypt unregistered [ 7295.570475] Key type ._llcrypt registered [ 7295.572863] Key type .llcrypt registered [ 7295.821625] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 7297.834970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7297.939490] alg: No test for adler32 (adler32-zlib) [ 7299.061978] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 7299.368953] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 7301.023454] Key type lgssc registered [ 7302.628869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7310.352580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7310.369980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7311.904359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7316.662586] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7322.088533] LustreError: 191626:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7322.121423] LustreError: 191626:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7325.419095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7325.783810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7329.928594] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7337.942359] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7343.815390] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7351.416775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7351.556500] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7351.558395] LustreError: 193492:0:(llog.c:1696:llog_backup()) MGC192.168.203.101@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7351.563321] Lustre: 193492:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.101@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7351.639470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7357.188691] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7364.547357] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7367.145622] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7367.154712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7367.159926] Lustre: Skipped 1 previous similar message [ 7371.298464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7371.310032] Lustre: Skipped 4 previous similar messages [ 7373.102815] Lustre: server umount lustre-MDT0000 complete [ 7376.357336] LustreError: 191627:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7376.381507] LustreError: 191627:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7376.474802] LustreError: 191606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066032 with bad export cookie 16425196674335698356 [ 7376.500698] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7377.504760] LustreError: 192358:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e1e63a58 x1875971622856960/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7377.742228] Lustre: server umount lustre-MDT0001 complete [ 7391.727109] Lustre: server umount lustre-OST0000 complete [ 7401.437556] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 7412.123343] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 7450.083908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7450.455887] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7450.505866] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7450.642723] Lustre: lustre-MDT0000: new disk, initializing [ 7450.734988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7450.765280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7455.632926] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7466.580634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7466.703351] Lustre: 197014:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7466.743625] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7466.746171] Lustre: Skipped 1 previous similar message [ 7466.748974] LustreError: 197798:0:(llog.c:1696:llog_backup()) MGC192.168.203.101@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7466.753210] LustreError: 197798:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7466.757074] Lustre: 197798:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.101@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7466.770360] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7466.774173] Lustre: Skipped 1 previous similar message [ 7466.829418] Lustre: lustre-MDT0001: new disk, initializing [ 7466.928476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7466.985766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7466.995752] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7471.222534] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7480.377903] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7485.747595] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7492.248216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7492.585479] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7492.587163] Lustre: Skipped 1 previous similar message [ 7492.593726] LustreError: 198959:0:(llog.c:1696:llog_backup()) MGC192.168.203.101@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7492.600143] LustreError: 198959:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7492.607950] Lustre: 198959:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.101@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7492.616431] Lustre: 198959:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7492.652353] Lustre: lustre-OST0000: new disk, initializing [ 7492.659562] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7492.669741] Lustre: 198983:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7492.785801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7499.039624] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7499.810284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7499.824070] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7499.926683] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7508.605387] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7513.542717] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7513.834470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7518.478987] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7518.665974] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7520.740627] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7520.753487] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7520.757101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7520.768909] Lustre: Skipped 2 previous similar messages [ 7520.788779] Lustre: Skipped 1 previous similar message [ 7525.857487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7525.867462] Lustre: Skipped 1 previous similar message [ 7526.891078] Lustre: server umount lustre-OST0000 complete [ 7530.977644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7530.985651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7530.992635] Lustre: Skipped 1 previous similar message [ 7531.008159] Lustre: Skipped 1 previous similar message [ 7536.749341] Lustre: server umount lustre-MDT0000 complete [ 7540.231268] LustreError: 197006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066196 with bad export cookie 16425196674335699077 [ 7540.247089] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7540.248874] LustreError: 197006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7540.571152] Lustre: server umount lustre-MDT0001 complete [ 7549.828691] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 14:50:04 (1789066204) [ 7567.730966] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 7576.490610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7576.786747] LustreError: 202109:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7576.828974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7581.516392] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7582.179494] LustreError: 202110:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7587.303895] LustreError: 202109:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7591.742680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7598.243885] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7601.605539] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7608.144283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7614.487530] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7623.489188] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7623.651833] Lustre: lustre-OST0001: new disk, initializing [ 7623.657633] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7623.662882] Lustre: 204419:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7623.751115] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7623.755122] Lustre: Skipped 2 previous similar messages [ 7629.964832] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7632.534257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7632.548568] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7632.656216] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7641.587246] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7653.092602] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7655.043641] Lustre: DEBUG MARKER: ADD OST3 [ 7659.783223] loop: module loaded [ 7659.886712] loop0: detected capacity change from 0 to 204800000 [ 7660.072052] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7660.091187] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7660.116735] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7666.433574] loop0: detected capacity change from 0 to 204800000 [ 7667.255911] Lustre: DEBUG MARKER: STOP OST3 [ 7671.962510] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7672.143671] Lustre: lustre-OST0003: new disk, initializing [ 7672.153536] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7672.171137] Lustre: 206439:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7673.499555] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7673.736653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7673.746440] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7673.790628] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7675.551138] LustreError: 206557:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7675.803624] Lustre: Failing over lustre-OST0003 [ 7676.037531] Lustre: server umount lustre-OST0003 complete [ 7678.957168] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7678.967299] LustreError: Skipped 1 previous similar message [ 7678.970106] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7678.985286] LustreError: 203595:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7679.011994] LustreError: 203595:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7679.985617] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7687.536665] Lustre: DEBUG MARKER: START OST3 again [ 7694.032332] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7694.324721] Lustre: lustre-OST0003: Not available for connect from 0@lo (not set up) [ 7694.336269] Lustre: Skipped 3 previous similar messages [ 7694.527670] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7694.548378] Lustre: Skipped 1 previous similar message [ 7694.568743] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7694.850376] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7696.263692] Lustre: lustre-OST0003: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 7696.269142] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7702.617474] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7723.440604] Lustre: Failing over lustre-OST0003 [ 7723.492963] Lustre: server umount lustre-OST0003 complete [ 7725.551702] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7725.564486] Lustre: Skipped 1 previous similar message [ 7725.575574] LustreError: 203597:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7725.604857] LustreError: 203597:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 7740.897254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7740.899610] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7740.906734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7740.946740] Lustre: Skipped 4 previous similar messages [ 7746.786541] Lustre: server umount lustre-MDT0000 complete [ 7750.674890] LustreError: 202091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066407 with bad export cookie 16425196674335699959 [ 7750.692553] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7750.695316] LustreError: 202091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7757.655907] Lustre: server umount lustre-MDT0001 complete [ 7762.723722] Lustre: server umount lustre-OST0000 complete [ 7767.007875] Lustre: server umount lustre-OST0001 complete [ 7775.761346] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 7786.011153] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 7825.193343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7825.507847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7825.572679] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7825.701655] Lustre: lustre-MDT0000: new disk, initializing [ 7825.788630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7825.815291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7830.808579] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7841.051123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7841.169562] Lustre: 211878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7841.503804] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7845.920227] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7855.033672] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7861.478197] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7868.240488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7868.515619] Lustre: lustre-OST0000: new disk, initializing [ 7868.518635] Lustre: Skipped 1 previous similar message [ 7868.523894] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7868.530524] Lustre: Skipped 2 previous similar messages [ 7868.537768] Lustre: 213848:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7869.751030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7869.759496] Lustre: Skipped 1 previous similar message [ 7869.762826] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7869.880455] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7875.006722] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7885.718321] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7890.932538] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7891.212730] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7895.863647] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7896.119564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7900.641720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7900.654763] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7900.660254] Lustre: Skipped 2 previous similar messages [ 7900.665884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7900.680197] Lustre: Skipped 12 previous similar messages [ 7904.152102] Lustre: server umount lustre-OST0000 complete [ 7914.271541] Lustre: server umount lustre-MDT0000 complete [ 7916.003845] LustreError: 211886:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7916.022578] LustreError: 211886:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 7918.017994] LustreError: 211872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066574 with bad export cookie 16425196674335702255 [ 7918.028589] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7918.042244] LustreError: 211872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7918.361184] Lustre: server umount lustre-MDT0001 complete [ 7921.994471] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 14:56:16 (1789066576) [ 7936.624887] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 7947.486951] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 7988.245707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7988.627139] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7988.660125] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7988.754577] Lustre: lustre-MDT0000: new disk, initializing [ 7988.817902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7994.167197] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8005.383501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8005.568196] Lustre: 218460:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8005.910690] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8009.504750] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8017.390271] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8022.580094] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8029.272181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8029.564243] Lustre: 220427:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8031.318590] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8035.509993] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8044.381136] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8049.765404] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8050.083968] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8056.303512] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8056.805399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8061.934278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8061.941301] LustreError: Skipped 1 previous similar message [ 8061.950110] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8061.951233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8061.962150] Lustre: Skipped 4 previous similar messages [ 8061.993217] Lustre: Skipped 3 previous similar messages [ 8065.857446] Lustre: server umount lustre-OST0000 complete [ 8076.434389] Lustre: server umount lustre-MDT0000 complete [ 8077.285123] LustreError: 218471:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8077.315285] LustreError: 218471:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8080.120307] LustreError: 221969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066736 with bad export cookie 16425196674335703137 [ 8080.121582] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8080.135595] LustreError: 221969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8080.534237] Lustre: server umount lustre-MDT0001 complete [ 8089.564487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8090.149808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8090.159618] Lustre: Skipped 5 previous similar messages [ 8095.833653] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8104.288317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8108.902514] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8115.693284] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8120.778126] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8127.502765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8134.169830] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8141.761387] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8154.592402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8154.607097] LustreError: Skipped 1 previous similar message [ 8154.620292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8154.639884] Lustre: Skipped 2 previous similar messages [ 8165.343203] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8165.608531] Lustre: server umount lustre-OST0000 complete [ 8170.977117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8170.988397] LustreError: Skipped 1 previous similar message [ 8176.614100] Lustre: server umount lustre-MDT0000 complete [ 8180.798121] LustreError: 222626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066837 with bad export cookie 16425196674335704012 [ 8180.802260] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8180.823669] LustreError: 222626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8181.049133] Lustre: server umount lustre-MDT0001 complete [ 8281.405073] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 8284.063499] Key type lgssc unregistered [ 8284.387135] LNet: 226316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8284.394628] LNetError: 226316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8284.408365] LNet: Removed LNI 192.168.203.101@tcp [ 8285.247141] Key type .llcrypt unregistered [ 8285.248624] Key type ._llcrypt unregistered [ 8310.925845] Key type ._llcrypt registered [ 8310.928757] Key type .llcrypt registered [ 8311.072726] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 8326.962233] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 8328.087261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8328.111769] alg: No test for adler32 (adler32-zlib) [ 8329.209701] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 8329.481452] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 8331.175348] Key type lgssc registered [ 8332.479177] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8364.896434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8364.935869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8366.206978] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8366.235290] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8366.308380] Lustre: lustre-MDT0000: new disk, initializing [ 8366.379214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8366.401507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8371.428275] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8382.205321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8382.308447] Lustre: 229668:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8382.366828] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8382.370879] Lustre: Skipped 1 previous similar message [ 8382.448060] Lustre: lustre-MDT0001: new disk, initializing [ 8382.554985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8382.597375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8382.603991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8387.050433] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8394.653715] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8398.768933] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8404.523590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8404.820622] Lustre: lustre-OST0000: new disk, initializing [ 8404.826114] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8404.838149] Lustre: 231637:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8404.929155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8410.564964] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8415.292823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8415.299532] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8415.449489] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8419.293166] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8424.280813] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8424.550791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8429.523588] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8429.711941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8436.194504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8436.198714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8436.208522] LustreError: Skipped 1 previous similar message [ 8436.214506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8436.247859] Lustre: Skipped 1 previous similar message [ 8437.773359] Lustre: server umount lustre-OST0000 complete [ 8441.313518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8441.326340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8441.331570] Lustre: Skipped 2 previous similar messages [ 8446.433723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8446.442261] Lustre: Skipped 1 previous similar message [ 8447.527875] Lustre: server umount lustre-MDT0000 complete [ 8450.762057] LustreError: 229660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067107 with bad export cookie 12207750423512890563 [ 8450.775443] LustreError: 229660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8450.776108] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8451.038771] Lustre: server umount lustre-MDT0001 complete [ 8466.011705] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 8474.007788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8474.401641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8478.338975] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8479.713078] LustreError: 234567:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8479.733680] LustreError: 234567:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8483.812441] LustreError: 234568:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8486.356393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8486.795945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8491.242122] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8494.282698] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8500.313468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8500.670651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8506.302751] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8513.671869] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8513.811905] Lustre: lustre-OST0001: new disk, initializing [ 8513.817188] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8513.823146] Lustre: 236873:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8519.869957] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8521.326834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8521.344703] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8521.440554] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8530.579951] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8539.286582] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8551.437890] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 15:06:45 (1789067205) [ 8557.542640] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8557.557701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8557.584688] Lustre: Skipped 3 previous similar messages [ 8557.600989] Lustre: Skipped 3 previous similar messages [ 8559.930440] Lustre: server umount lustre-MDT0000 complete [ 8562.671169] LustreError: 234562:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8562.697994] LustreError: 234562:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8563.750138] LustreError: 234549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067220 with bad export cookie 12207750423512891445 [ 8563.752260] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8563.769650] LustreError: 234549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8564.282569] Lustre: server umount lustre-MDT0001 complete [ 8578.147124] Lustre: server umount lustre-OST0000 complete [ 8592.329405] Lustre: server umount lustre-OST0001 complete [ 8598.680587] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 8611.362454] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 8645.915307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8646.246586] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8646.265283] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8646.343528] Lustre: lustre-MDT0000: new disk, initializing [ 8646.419057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8646.424201] Lustre: Skipped 1 previous similar message [ 8646.438253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8651.090497] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8661.462950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8661.657954] Lustre: 240817:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8661.728220] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8661.731170] Lustre: Skipped 1 previous similar message [ 8661.835571] Lustre: lustre-MDT0001: new disk, initializing [ 8661.977050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8661.985950] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8666.568131] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8675.156264] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8681.203702] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8689.993977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8690.466534] Lustre: lustre-OST0000: new disk, initializing [ 8690.469893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8690.475068] Lustre: 242788:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8690.576807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8690.581341] Lustre: Skipped 1 previous similar message [ 8692.277663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8692.292608] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8692.527368] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8697.942801] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8706.776614] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8711.595098] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8711.803648] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8716.526887] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8716.715814] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8723.429267] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8723.443929] Lustre: Skipped 1 previous similar message [ 8723.449289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8723.452350] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8723.461123] Lustre: Skipped 1 previous similar message [ 8724.758158] Lustre: server umount lustre-OST0000 complete [ 8728.545203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8728.545957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8728.572919] Lustre: Skipped 2 previous similar messages [ 8733.666916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8733.675446] Lustre: Skipped 3 previous similar messages [ 8734.651321] Lustre: server umount lustre-MDT0000 complete [ 8737.708709] LustreError: 240811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067394 with bad export cookie 12207750423512893580 [ 8737.725214] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8738.107581] Lustre: server umount lustre-MDT0001 complete [ 8745.498695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8745.897676] LustreError: 245005:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8750.158331] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8751.083854] LustreError: 245006:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8757.749319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8758.044727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8758.049789] Lustre: Skipped 1 previous similar message [ 8761.310895] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8766.562830] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8770.237206] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8776.933830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8784.153986] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 8791.669419] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8934.882691] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8934.900722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8934.901349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8934.913650] Lustre: Skipped 1 previous similar message [ 8949.215138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8949.361435] Lustre: server umount lustre-OST0000 complete [ 8952.802773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8952.811393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8952.826947] Lustre: Skipped 1 previous similar message [ 8958.738257] Lustre: server umount lustre-MDT0000 complete [ 8961.596291] LustreError: 245740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067617 with bad export cookie 12207750423512894462 [ 8961.603128] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8961.609692] LustreError: 245740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8961.858568] Lustre: server umount lustre-MDT0001 complete [ 9058.993473] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 9061.442274] Key type lgssc unregistered [ 9061.707555] LNet: 248612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9061.714881] LNetError: 248612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9061.738375] LNet: Removed LNI 192.168.203.101@tcp [ 9062.621211] Key type .llcrypt unregistered [ 9062.624061] Key type ._llcrypt unregistered [ 9082.459342] Key type ._llcrypt registered [ 9082.461932] Key type .llcrypt registered [ 9082.540720] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 9094.256087] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 9095.082949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9095.201444] alg: No test for adler32 (adler32-zlib) [ 9096.202847] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 9096.433381] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 9098.119196] Key type lgssc registered [ 9099.104412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9136.981862] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 9146.040707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9146.066826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9147.334583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9147.357296] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9147.440210] Lustre: lustre-MDT0000: new disk, initializing [ 9147.528476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9147.563508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9151.927580] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9161.451345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9161.586608] Lustre: 252656:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9161.673305] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9161.681419] Lustre: Skipped 1 previous similar message [ 9161.765486] Lustre: lustre-MDT0001: new disk, initializing [ 9161.893491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9161.949071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9161.967486] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9165.716955] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9169.960136] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9175.267790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9175.482521] Lustre: lustre-OST0000: new disk, initializing [ 9175.485601] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9175.490918] Lustre: 254286:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9175.572496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9177.595145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9177.601977] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9177.737200] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9180.776480] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9190.876424] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9191.166178] Lustre: lustre-OST0001: new disk, initializing [ 9191.169672] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9191.176897] Lustre: 255156:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9191.310560] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9198.204947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9198.226065] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9198.330463] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9198.376502] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9209.418370] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9219.563969] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9233.898634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9233.906667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9233.920827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9234.406240] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9234.436585] Lustre: Skipped 1 previous similar message [ 9234.441984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9234.444658] Lustre: Skipped 3 previous similar messages [ 9237.631804] Lustre: server umount lustre-MDT0000 complete [ 9241.362787] LustreError: 252647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067897 with bad export cookie 2747335030755238834 [ 9241.370154] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9241.383186] LustreError: 252647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9241.797642] Lustre: server umount lustre-MDT0001 complete [ 9255.938831] Lustre: server umount lustre-OST0000 complete [ 9259.999245] Lustre: 250162:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789067900/real 1789067900] req@ffff91db87388700 x1875973507198592/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789067916 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9260.039856] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9260.053465] Lustre: Skipped 1 previous similar message [ 9260.657169] Lustre: server umount lustre-OST0001 complete [ 9275.517227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9284.887040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9293.358332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9302.637088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9321.319747] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 9330.994451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9331.029440] Lustre: lustre-MDT0000: reset Object Index mappings [ 9331.387570] LustreError: 260167:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9331.397311] LustreError: 260167:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 9331.443761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9336.392973] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9336.802879] LustreError: 260168:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9340.905813] LustreError: 260167:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9345.388563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9345.405436] Lustre: lustre-MDT0001: reset Object Index mappings [ 9345.835930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9350.430200] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9353.012060] Lustre: 261308:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9358.948729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9364.745383] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9371.684307] LustreError: 261661:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9371.859291] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9372.045717] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9372.050098] Lustre: Skipped 1 previous similar message [ 9376.179530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9376.185088] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9378.038493] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9384.591084] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9387.575866] Lustre: 263181:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9392.099968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9392.116538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9392.139798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9396.704434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9396.708872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9396.712170] Lustre: Skipped 1 previous similar message [ 9396.756691] Lustre: Skipped 1 previous similar message [ 9398.179533] Lustre: server umount lustre-MDT0000 complete [ 9401.834659] LustreError: 260164:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9401.863838] LustreError: 260164:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9401.926119] LustreError: 260148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789068058 with bad export cookie 2747335030755245624 [ 9401.932739] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9401.939707] LustreError: 260148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9402.335330] Lustre: server umount lustre-MDT0001 complete [ 9416.293583] Lustre: server umount lustre-OST0000 complete [ 9428.912177] Lustre: server umount lustre-OST0001 complete [ 9445.957591] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 15:21:40 (1789068100) [ 9461.051863] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 9469.671508] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 9501.013205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9501.238799] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9501.280698] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9501.352410] Lustre: lustre-MDT0000: new disk, initializing [ 9501.454491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9501.479912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9505.835924] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9514.346862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9514.413925] Lustre: 267434:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9514.471716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9514.475041] Lustre: Skipped 1 previous similar message [ 9514.571978] Lustre: lustre-MDT0001: new disk, initializing [ 9514.709383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9514.720443] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9518.840990] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9526.468575] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9530.829626] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9536.840730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9537.123253] Lustre: lustre-OST0000: new disk, initializing [ 9537.127581] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9537.142732] Lustre: 269402:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9538.241113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9538.260171] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9538.361913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9542.694673] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9551.371383] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9555.753508] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9555.979117] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9562.075922] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9562.271201] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9564.136065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9564.149080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9564.152133] Lustre: Skipped 1 previous similar message [ 9564.153233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9564.170889] Lustre: Skipped 3 previous similar messages [ 9570.305670] Lustre: server umount lustre-OST0000 complete [ 9574.372686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9574.377878] Lustre: Skipped 3 previous similar messages [ 9580.454045] Lustre: server umount lustre-MDT0000 complete [ 9583.898476] LustreError: 267428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789068240 with bad export cookie 2747335030755247717 [ 9583.911415] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9583.920222] LustreError: 267428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9584.282798] Lustre: server umount lustre-MDT0001 complete [ 9602.382617] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 9611.131283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9611.502495] LustreError: 272334:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9611.548200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9611.557167] Lustre: Skipped 2 previous similar messages [ 9615.604986] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9623.317251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9628.624354] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9631.799976] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9638.133568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9645.275489] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9653.116484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9653.363866] Lustre: lustre-OST0001: new disk, initializing [ 9653.371310] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9653.380816] Lustre: 274642:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9654.602339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9654.611591] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9654.722914] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9659.241850] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9667.567289] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9670.189288] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9674.108680] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9674.125080] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9684.964503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9684.972035] LustreError: Skipped 1 previous similar message [ 9684.978040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9684.992032] Lustre: Skipped 3 previous similar messages [ 9685.004822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9685.015881] Lustre: Skipped 2 previous similar messages [ 9689.043049] Lustre: server umount lustre-MDT0000 complete [ 9691.625573] LustreError: 272331:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9691.648070] LustreError: 272331:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 9692.831498] LustreError: 272316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789068349 with bad export cookie 2747335030755248599 [ 9692.836141] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9692.839520] LustreError: 272316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9693.146198] Lustre: server umount lustre-MDT0001 complete [ 9707.100747] Lustre: server umount lustre-OST0000 complete [ 9721.422861] Lustre: server umount lustre-OST0001 complete [ 9739.814930] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 9750.052039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9750.424463] LustreError: 277985:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9750.497051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9750.501921] Lustre: Skipped 3 previous similar messages [ 9755.180155] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9764.025908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9768.861536] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9771.631134] Lustre: 279127:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9778.254829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9784.481888] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9793.109213] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9795.453979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9795.521485] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9798.956861] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9806.619533] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9809.799481] Lustre: 281004:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9813.995661] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9814.104701] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9826.311849] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 15:28:00 (1789068480) [ 9841.120036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9841.130677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9841.140168] Lustre: Skipped 3 previous similar messages [ 9841.148104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9841.152488] Lustre: Skipped 4 previous similar messages [ 9847.525381] Lustre: server umount lustre-MDT0000 complete [ 9850.969984] LustreError: 277966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789068507 with bad export cookie 2747335030755254969 [ 9850.971821] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9850.985498] LustreError: 277966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9851.875344] LustreError: 277981:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9851.889501] LustreError: 277981:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 9857.401318] Lustre: server umount lustre-MDT0001 complete [ 9867.316785] Lustre: server umount lustre-OST0000 complete [ 9871.685539] Lustre: server umount lustre-OST0001 complete [ 9878.256298] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 9889.221819] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 9932.460118] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 9943.791774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9944.076948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9944.126115] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9944.234404] Lustre: lustre-MDT0000: new disk, initializing [ 9944.403193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9949.866988] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9960.696040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9960.852482] Lustre: 285536:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9961.085927] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9964.948969] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9976.361511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9976.628873] Lustre: 287168:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9978.510348] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9981.952714] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 9992.249299] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9992.468273] Lustre: 288039:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9993.780888] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9993.792359] Lustre: Skipped 1 previous similar message [ 9993.942554] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9999.635524] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10009.363665] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10013.300885] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10013.304462] Lustre: Skipped 1 previous similar message [10031.820436] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 15:31:26 (1789068686) [10035.174924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10035.184799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10035.197241] Lustre: Skipped 6 previous similar messages [10035.201348] Lustre: Skipped 11 previous similar messages [10040.356930] LustreError: 289206:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff91dcbe0c7700) refcount nonzero (1) after lock cleanup; forcing cleanup. [10040.419581] Lustre: server umount lustre-MDT0000 complete [10044.313900] LustreError: 285528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789068700 with bad export cookie 2747335030755262893 [10044.316842] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10044.332485] LustreError: 285528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10044.779939] Lustre: server umount lustre-MDT0001 complete [10059.510552] Lustre: server umount lustre-OST0000 complete [10061.413480] Lustre: 250162:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789068701/real 1789068701] req@ffff91db8a318a80 x1875973507492224/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789068717 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10064.039522] Lustre: server umount lustre-OST0001 complete [10073.285609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10073.788788] LustreError: 290288:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10073.811073] LustreError: 290288:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10073.922186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10073.934491] Lustre: Skipped 7 previous similar messages [10079.038968] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10087.305046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10092.394802] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10099.898283] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10105.677236] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10112.982298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10119.933560] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10126.732243] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10145.291567] loop1: detected capacity change from 0 to 204800000 [10145.546725] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10145.557929] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10145.571136] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10150.510421] loop1: detected capacity change from 0 to 204800000 [10156.222878] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10156.287912] LustreError: 290275:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs18151: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10156.304490] LustreError: 290275:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs18151 [10156.316631] LustreError: 290275:0:(mgs_handler.c:517:mgs_target_reg()) fs18151-MDT0001: Error checking target index: rc = -13 [10156.324954] LustreError: fs18151-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10156.332997] LustreError: 294008:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10156.339792] LustreError: 294008:0:(tgt_mount.c:2128:server_put_super()) no obd fs18151-MDT0001 [10156.344962] LustreError: 294008:0:(tgt_mount.c:116:server_deregister_mount()) fs18151-MDT0001 not registered [10156.370685] Lustre: server umount fs18151-MDT0001 complete [10156.373693] LustreError: 294008:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10164.822450] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10164.911915] Lustre: Setting parameter fs18151-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs18151-MDT0001 [10164.967944] Lustre: srv-fs18151-MDT0001: No data found on store. Initialize space. [10164.972256] Lustre: Skipped 4 previous similar messages [10165.057803] Lustre: fs18151-MDT0001: new disk, initializing [10165.063374] Lustre: Skipped 3 previous similar messages [10165.187490] Lustre: srv-fs18151-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10165.194721] Lustre: 294475:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs18151-MDT0001-mdtlov: can't create new llog: rc = -115 [10165.205200] LustreError: 294475:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs18151-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10169.881346] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10175.751182] Lustre: server umount fs18151-MDT0001 complete [10188.058836] loop1: detected capacity change from 0 to 204800000 [10193.348396] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10193.599901] Lustre: 296044:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs18151-MDT0001-mdtlov: can't create new llog: rc = -115 [10193.613773] LustreError: 296044:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs18151-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10197.276061] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10198.504476] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10201.416491] Lustre: server umount fs18151-MDT0001 complete [10213.860881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10213.882784] Lustre: Skipped 3 previous similar messages [10213.885323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10213.898374] Lustre: Skipped 6 previous similar messages [10217.941696] Lustre: server umount lustre-MDT0000 complete [10225.286639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10225.512234] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10230.168628] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10231.287782] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10237.650911] loop1: detected capacity change from 0 to 204800000 [10243.217081] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10243.671548] Lustre: 298488:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs18151-MDT0001-mdtlov: can't create new llog: rc = -115 [10243.691864] LustreError: 298488:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs18151-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10247.604560] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10252.656875] Lustre: server umount fs18151-MDT0001 complete [10275.296654] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10275.311793] LustreError: Skipped 1 previous similar message [10280.336809] Lustre: server umount lustre-OST0000 complete [10298.335241] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10298.613933] Lustre: server umount lustre-MDT0000 complete [10302.254276] LustreError: 290269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789068958 with bad export cookie 2747335030755265574 [10302.261607] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10302.268983] LustreError: 290269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10302.629993] Lustre: server umount lustre-MDT0001 complete [10315.243485] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [10318.110276] Key type lgssc unregistered [10318.447081] LNet: 300833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10318.459798] LNetError: 300833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10318.472389] LNet: Removed LNI 192.168.203.101@tcp [10319.350260] Key type .llcrypt unregistered [10319.352974] Key type ._llcrypt unregistered [10338.645777] Key type ._llcrypt registered [10338.648696] Key type .llcrypt registered [10338.764246] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [10339.761245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10339.910343] alg: No test for adler32 (adler32-zlib) [10341.127597] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [10341.415994] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [10343.071208] Key type lgssc registered [10344.011791] Lustre: Echo OBD driver; http://www.lustre.org/ [10351.390644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10351.422982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10352.821182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10356.454446] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10363.181811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10363.364698] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10363.592164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10367.623943] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10374.930279] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10381.492989] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10389.777143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10390.175535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10397.241595] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10398.373287] LustreError: 303931:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10398.382304] LustreError: 303931:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10403.828437] LustreError: 303933:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10404.975797] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10407.656108] LustreError: 303933:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10413.038536] LustreError: 303931:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10413.059300] LustreError: 303931:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10418.153565] LustreError: 303933:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10418.173036] LustreError: 303933:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10427.445657] loop1: detected capacity change from 0 to 204800000 [10427.620322] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10427.629228] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10427.637639] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10428.399909] LustreError: 303931:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10428.431767] LustreError: 303931:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10433.021535] loop1: detected capacity change from 0 to 204800000 [10438.381153] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10438.522788] Lustre: fs23608-OST0000: new disk, initializing [10438.526415] Lustre: srv-fs23608-OST0000: No data found on store. Initialize space. [10438.532131] Lustre: 305842:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10438.612770] Lustre: fs23608-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10444.811688] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10448.885784] LustreError: 303931:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10448.903136] LustreError: 303931:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [10452.952338] Lustre: server umount fs23608-OST0000 complete [10464.756076] loop1: detected capacity change from 0 to 204800000 [10469.775852] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10470.129353] Lustre: fs23608-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10476.401339] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10479.605959] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10484.709961] LustreError: 303933:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10484.729103] LustreError: 303933:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [10485.744606] loop2: detected capacity change from 0 to 204800000 [10485.885264] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10485.894829] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10485.910312] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10489.846731] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10491.148088] loop2: detected capacity change from 0 to 204800000 [10497.164225] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10497.258473] LustreError: fs23608-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10497.267360] LustreError: fs23608-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10497.277415] LustreError: 308883:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10497.282661] LustreError: 308883:0:(tgt_mount.c:2128:server_put_super()) no obd fs23608-OST0001 [10497.286894] LustreError: 308883:0:(tgt_mount.c:116:server_deregister_mount()) fs23608-OST0001 not registered [10497.320281] Lustre: server umount fs23608-OST0001 complete [10497.326987] LustreError: 308883:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10507.098780] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10507.216820] Lustre: fs23608-OST0001: new disk, initializing [10507.219798] Lustre: srv-fs23608-OST0001: No data found on store. Initialize space. [10507.224943] Lustre: 309433:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10507.298092] Lustre: fs23608-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10513.885915] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10522.987819] Lustre: server umount fs23608-OST0000 complete [10532.689313] Lustre: server umount fs23608-OST0001 complete [10545.826805] loop1: detected capacity change from 0 to 204800000 [10551.269751] LustreError: 304606:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10551.286291] LustreError: 304606:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [10552.568618] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10552.804314] Lustre: fs23608-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10559.266980] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10565.827747] loop2: detected capacity change from 0 to 204800000 [10570.554916] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10577.406935] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10586.795727] Lustre: server umount fs23608-OST0000 complete [10597.658831] Lustre: server umount fs23608-OST0001 complete [10612.710948] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10612.741249] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10612.741875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10612.762800] Lustre: Skipped 1 previous similar message [10615.630918] Lustre: server umount lustre-OST0000 complete [10619.873428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10619.881758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10619.891961] Lustre: Skipped 1 previous similar message [10619.897704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10622.950747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10622.953265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10625.254821] Lustre: server umount lustre-MDT0000 complete [10628.883188] LustreError: 302081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789069285 with bad export cookie 12501965880249119402 [10628.885736] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10628.902752] LustreError: 302081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10629.435605] Lustre: server umount lustre-MDT0001 complete [10640.181518] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [10642.832900] Key type lgssc unregistered [10643.160167] LNet: 315356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10643.171512] LNetError: 315356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10644.196739] LNet: Removed LNI 192.168.203.101@tcp [10645.033272] Key type .llcrypt unregistered [10645.039117] Key type ._llcrypt unregistered [10671.841915] Key type ._llcrypt registered [10671.843654] Key type .llcrypt registered [10671.909835] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [10688.049212] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [10689.126639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10689.159054] alg: No test for adler32 (adler32-zlib) [10690.260706] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [10690.477814] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [10692.143193] Key type lgssc registered [10693.300775] Lustre: Echo OBD driver; http://www.lustre.org/ [10730.480037] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10730.529805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10731.773546] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10731.795112] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10731.859930] Lustre: lustre-MDT0000: new disk, initializing [10731.961748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10731.975311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10736.021973] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10747.489949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10747.687527] Lustre: 319211:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10747.825798] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10747.837273] Lustre: Skipped 1 previous similar message [10747.966557] Lustre: lustre-MDT0001: new disk, initializing [10748.086960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10748.171944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10748.195593] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10754.234084] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10763.159926] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10767.901435] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10773.710276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10774.010115] Lustre: lustre-OST0000: new disk, initializing [10774.015409] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10774.022298] Lustre: 321177:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10774.120057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10779.709076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10779.728347] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10779.833478] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10780.539493] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [10789.299062] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10792.040238] Lustre: Mounted lustre-client - version 2.17.58_39_gf941a14 [10793.126407] Lustre: Unmounted lustre-client [10807.780418] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10807.810978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10807.818757] Lustre: Skipped 1 previous similar message [10812.157733] Lustre: server umount lustre-OST0000 complete [10818.015976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10818.019845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10818.029521] Lustre: Skipped 2 previous similar messages [10818.056114] Lustre: Skipped 1 previous similar message [10821.993841] Lustre: server umount lustre-MDT0000 complete [10825.878028] LustreError: 319203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789069482 with bad export cookie 11122951913471703109 [10825.886317] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10825.890730] LustreError: 319203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10826.521420] Lustre: server umount lustre-MDT0001 complete [10840.899170] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [10844.685717] Key type lgssc unregistered [10846.119789] LNet: 323338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10846.138487] LNetError: 323338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10920.945825] LNet: Removed LNI 192.168.203.101@tcp [10922.342771] Key type .llcrypt unregistered [10922.350297] Key type ._llcrypt unregistered [10948.044856] Key type ._llcrypt registered [10948.046890] Key type .llcrypt registered [10948.206989] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [10966.331754] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [10968.175741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10968.207043] alg: No test for adler32 (adler32-zlib) [10969.422654] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [10969.668390] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [10971.407323] Key type lgssc registered [10972.965576] Lustre: Echo OBD driver; http://www.lustre.org/ [11005.259950] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 15:47:39 (1789069659) [11012.150285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11012.227990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11013.508628] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11013.560905] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11013.674356] Lustre: lustre-MDT0000: new disk, initializing [11013.758951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11013.798782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11018.162446] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11028.669919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11028.826260] Lustre: 326557:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11028.880534] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11028.888173] Lustre: Skipped 1 previous similar message [11028.976430] Lustre: lustre-MDT0001: new disk, initializing [11029.081095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11029.119726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11029.135603] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11033.786919] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11043.397906] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11049.752253] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11056.219516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11056.458210] Lustre: lustre-OST0000: new disk, initializing [11056.463649] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11056.471359] Lustre: 328526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11056.537060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11061.359726] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11064.980732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11065.004147] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11065.185086] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11071.773288] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11090.913544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11090.928438] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11090.933432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11090.968512] Lustre: Skipped 1 previous similar message [11096.033705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11096.058248] Lustre: Skipped 1 previous similar message [11096.402960] Lustre: server umount lustre-OST0000 complete [11100.640512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11100.666192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11100.696738] Lustre: Skipped 1 previous similar message [11100.700866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11106.191384] Lustre: server umount lustre-MDT0000 complete [11109.028851] LustreError: 326550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789069765 with bad export cookie 2275598786170785518 [11109.046351] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11109.054108] LustreError: 326550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11109.287351] Lustre: server umount lustre-MDT0001 complete [11117.104351] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [11119.986135] Key type lgssc unregistered [11120.329875] LNet: 330432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11120.342325] LNetError: 330432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11120.367667] LNet: Removed LNI 192.168.203.101@tcp [11121.303738] Key type .llcrypt unregistered [11121.306064] Key type ._llcrypt unregistered [11143.123052] Key type ._llcrypt registered [11143.128303] Key type .llcrypt registered [11143.262877] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [11144.448844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11144.545640] alg: No test for adler32 (adler32-zlib) [11145.755772] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [11146.040959] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [11147.783219] Key type lgssc registered [11149.110918] Lustre: Echo OBD driver; http://www.lustre.org/ [11157.983111] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11158.033805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11159.522034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11164.375898] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11168.740622] LustreError: 331704:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11168.750585] LustreError: 331704:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11173.427531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11173.876592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11178.578081] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11186.378479] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11192.406252] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11199.194477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11199.469898] LustreError: 333523:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11199.651803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11206.741622] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11214.949461] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11225.745856] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11227.281468] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11232.794039] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.1@tcp (stopping) [11234.786788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11234.795547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11234.799741] Lustre: Skipped 2 previous similar messages [11234.808416] Lustre: Skipped 2 previous similar messages [11237.363118] Lustre: server umount lustre-MDT0000 complete [11237.890171] LustreError: 331699:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11239.905265] LustreError: 333245:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11239.926108] LustreError: 333245:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11240.987031] LustreError: 331684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789069897 with bad export cookie 3822877841988098828 [11240.987947] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11240.992292] LustreError: 331684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11243.050428] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.1@tcp (stopping) [11245.025086] LustreError: 331701:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11245.025505] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11245.057976] LustreError: 331701:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11245.087190] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11247.462431] Lustre: server umount lustre-MDT0001 complete [11251.515428] Lustre: server umount lustre-OST0000 complete [11260.013458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11264.899217] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11267.696967] Lustre: 335886:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.101@tcp [11269.730880] Lustre: server umount MGS complete [11278.064259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11278.676707] LustreError: 336458:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11278.741386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11283.756316] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11292.927938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11293.245919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11298.438522] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11307.547131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11307.917423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11315.618203] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11324.666911] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11326.787044] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11333.374733] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11334.939653] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11340.796500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [11340.814797] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11340.828145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11341.876466] Lustre: server umount lustre-OST0000 complete [11349.485839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11349.506282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11349.546705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11351.616324] Lustre: server umount lustre-MDT0000 complete [11355.107753] LustreError: 338515:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11355.120489] LustreError: 338515:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11355.590724] LustreError: 336438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789070011 with bad export cookie 3822877841988099668 [11355.599669] LustreError: 336438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11355.599870] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11365.099608] Lustre: server umount lustre-MDT0001 complete [11378.187713] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [11380.857746] Key type lgssc unregistered [11381.250502] LNet: 339895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11381.260442] LNetError: 339895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11381.285282] LNet: Removed LNI 192.168.203.101@tcp [11382.471232] Key type .llcrypt unregistered [11382.473750] Key type ._llcrypt unregistered [11402.112782] Key type ._llcrypt registered [11402.118250] Key type .llcrypt registered [11402.285383] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [11418.982289] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [11419.957570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11420.017061] alg: No test for adler32 (adler32-zlib) [11421.200379] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [11421.551434] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [11423.327441] Key type lgssc registered [11424.766513] Lustre: Echo OBD driver; http://www.lustre.org/ [11460.243250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11460.284508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11461.590084] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11461.633534] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11461.727970] Lustre: lustre-MDT0000: new disk, initializing [11461.834141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11461.851695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11466.778690] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11477.532429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11477.659499] Lustre: 343055:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11477.732419] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11477.736762] Lustre: Skipped 1 previous similar message [11477.829677] Lustre: lustre-MDT0001: new disk, initializing [11477.929303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11477.956586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11477.969846] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11482.502697] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11491.922396] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11497.246967] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11503.636965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11503.845268] Lustre: lustre-OST0000: new disk, initializing [11503.849531] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11503.855481] Lustre: 345027:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11503.990578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11511.151741] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11512.906840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11512.925491] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11513.014666] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11520.066951] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11524.276246] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11524.536977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11529.901545] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11530.144405] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11533.792337] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11533.792650] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11533.795536] LustreError: Skipped 1 previous similar message [11533.801714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11533.815281] Lustre: Skipped 1 previous similar message [11533.830155] Lustre: Skipped 1 previous similar message [11538.249950] Lustre: server umount lustre-OST0000 complete [11544.035290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11544.036984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11544.050344] Lustre: Skipped 1 previous similar message [11548.539142] Lustre: server umount lustre-MDT0000 complete [11552.277463] LustreError: 345026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789070208 with bad export cookie 8759179432941249755 [11552.282234] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11552.290182] LustreError: 345026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11552.622866] Lustre: server umount lustre-MDT0001 complete [11560.896249] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 15:56:55 (1789070215) [11579.235529] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [11591.994200] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [11625.909290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11626.154885] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11626.177727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11626.279932] Lustre: lustre-MDT0000: new disk, initializing [11626.384854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11626.404276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11631.041578] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11641.214989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11641.332183] Lustre: 349786:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11641.380326] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11641.384781] Lustre: Skipped 1 previous similar message [11641.436903] Lustre: lustre-MDT0001: new disk, initializing [11641.501206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11641.551146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11641.556528] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11645.629120] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11655.147451] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11661.150444] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11668.006461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11668.297281] Lustre: lustre-OST0000: new disk, initializing [11668.301523] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11668.306753] Lustre: 351755:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11668.398626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11674.047208] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [11675.733510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11675.741051] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11675.902789] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11683.862408] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11714.253199] Lustre: server umount lustre-OST0000 complete [11718.624947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11718.639621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11718.670662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11718.676026] Lustre: Skipped 1 previous similar message [11722.211087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11722.214399] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11727.331051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11727.346870] Lustre: Skipped 2 previous similar messages [11728.441583] Lustre: server umount lustre-MDT0000 complete [11732.291270] LustreError: 349778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789070388 with bad export cookie 8759179432941250637 [11732.304207] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11732.312214] LustreError: 349778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11742.888479] Lustre: server umount lustre-MDT0001 complete [11755.743149] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [11759.610337] Key type lgssc unregistered [11759.917713] LNet: 353906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11759.937977] LNetError: 353906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11761.001448] LNet: Removed LNI 192.168.203.101@tcp [11761.759953] Key type .llcrypt unregistered [11761.762053] Key type ._llcrypt unregistered [11853.905861] Key type ._llcrypt registered [11853.910210] Key type .llcrypt registered [11854.032430] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [11854.364528] Key type .llcrypt unregistered [11854.368785] Key type ._llcrypt unregistered [11925.885403] Key type ._llcrypt registered [11925.890110] Key type .llcrypt registered [11926.035957] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [11926.437367] Key type .llcrypt unregistered [11926.443425] Key type ._llcrypt unregistered [11959.027178] Key type ._llcrypt registered [11959.029277] Key type .llcrypt registered [11959.111793] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [11959.338632] Key type .llcrypt unregistered [11959.342937] Key type ._llcrypt unregistered [11988.648933] Key type ._llcrypt registered [11988.651933] Key type .llcrypt registered [11988.761944] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [11989.061274] Key type .llcrypt unregistered [11989.063278] Key type ._llcrypt unregistered [12068.562736] Key type ._llcrypt registered [12068.569807] Key type .llcrypt registered [12068.787261] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [12069.184355] Key type .llcrypt unregistered [12069.189869] Key type ._llcrypt unregistered [12102.187230] Key type ._llcrypt registered [12102.190325] Key type .llcrypt registered [12102.299927] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [12102.635574] Key type .llcrypt unregistered [12102.639777] Key type ._llcrypt unregistered [12129.990444] Key type ._llcrypt registered [12129.992164] Key type .llcrypt registered [12130.084897] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [12130.323496] Key type .llcrypt unregistered [12130.328711] Key type ._llcrypt unregistered [12161.943688] Key type ._llcrypt registered [12161.947915] Key type .llcrypt registered [12162.092925] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [12162.401311] Key type .llcrypt unregistered [12162.402813] Key type ._llcrypt unregistered [12194.361682] Key type ._llcrypt registered [12194.363356] Key type .llcrypt registered [12194.508929] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [12194.925567] Key type .llcrypt unregistered [12194.927738] Key type ._llcrypt unregistered [12227.257852] Key type ._llcrypt registered [12227.260272] Key type .llcrypt registered [12227.396682] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [12227.703455] Key type .llcrypt unregistered [12227.705579] Key type ._llcrypt unregistered [12292.719303] Key type ._llcrypt registered [12292.721563] Key type .llcrypt registered [12292.837413] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [12293.200608] Key type .llcrypt unregistered [12293.202893] Key type ._llcrypt unregistered [12328.791575] Key type ._llcrypt registered [12328.793698] Key type .llcrypt registered [12328.946058] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [12329.388693] Key type .llcrypt unregistered [12329.392146] Key type ._llcrypt unregistered [12363.734269] Key type ._llcrypt registered [12363.736021] Key type .llcrypt registered [12363.869142] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [12364.313358] Key type .llcrypt unregistered [12364.315875] Key type ._llcrypt unregistered [12423.442524] Key type ._llcrypt registered [12423.445063] Key type .llcrypt registered [12423.561537] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [12442.249544] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [12443.432331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12443.473947] alg: No test for adler32 (adler32-zlib) [12444.673645] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [12444.957429] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [12446.633636] Key type lgssc registered [12447.981632] Lustre: Echo OBD driver; http://www.lustre.org/ [12489.884584] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12489.925593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12491.281028] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12491.322615] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12491.404185] Lustre: tst250fs-MDT0000: new disk, initializing [12491.480110] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12491.499281] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12496.915844] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12508.265443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12508.396792] Lustre: 374189:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12508.474922] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12508.478031] Lustre: Skipped 1 previous similar message [12508.582777] Lustre: tst250fs-MDT0001: new disk, initializing [12508.680835] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12508.733157] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12508.751788] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12513.274984] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12519.173070] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12525.279944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12525.650505] Lustre: tst250fs-OST0000: new disk, initializing [12525.654731] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12525.661858] Lustre: 375822:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12525.754406] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12534.837442] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12534.982716] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12535.011778] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12535.221842] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12547.516179] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12547.810376] Lustre: tst250fs-OST0001: new disk, initializing [12547.817634] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12547.826878] Lustre: 376695:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12548.117807] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12553.960106] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12553.975270] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12554.184560] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12556.566907] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [12564.227558] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12566.352920] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [12566.357687] Lustre: Skipped 1 previous similar message [12590.053446] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12590.064519] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12590.080146] Lustre: Skipped 2 previous similar messages [12590.094488] Lustre: Skipped 1 previous similar message [12595.176083] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12595.210305] Lustre: Skipped 5 previous similar messages [12600.298691] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12600.313678] Lustre: Skipped 3 previous similar messages [12601.311225] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12601.701215] Lustre: server umount tst250fs-MDT0000 complete [12606.915543] LustreError: 374179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789071263 with bad export cookie 10343262070299158650 [12606.923582] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12606.931236] LustreError: 374179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12607.384988] Lustre: server umount tst250fs-MDT0001 complete [12622.937063] Lustre: server umount tst250fs-OST0000 complete [12625.695562] Lustre: 372411:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789071266/real 1789071266] req@ffff91db868d3100 x1875977018882048/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789071282 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12625.713329] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12625.726220] Lustre: Skipped 1 previous similar message [12626.911315] Lustre: 372409:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789071267/real 1789071267] req@ffff91dc85bb7100 x1875977018882304/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789071283 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12627.268835] Lustre: server umount tst250fs-OST0001 complete [12647.984804] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [12657.443320] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [12693.302536] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 16:15:47 (1789071347) [12695.674515] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12715.863610] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [12726.254109] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [12762.218540] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12458 sec ========= 16:16:55 (1789071415) [12764.196499] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:16:58 (1789071418) === [12768.133918] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:17:02 (1789071422) === [12818.418774] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [12822.548892] Key type lgssc unregistered [12823.086860] LNet: 385836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12823.110332] LNetError: 385836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12823.146985] LNet: Removed LNI 192.168.203.101@tcp [12824.366226] Key type .llcrypt unregistered [12824.369274] Key type ._llcrypt unregistered