[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 468858556 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002385] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008015] pid_max: default: 32768 minimum: 301 [ 0.009139] LSM: Security Framework initializing [ 0.010050] Yama: becoming mindful. [ 0.011031] SELinux: Initializing. [ 0.011838] *** VALIDATE selinux *** [ 0.019779] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024181] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025159] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029120] *** VALIDATE tmpfs *** [ 0.030520] *** VALIDATE proc *** [ 0.032208] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.035091] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036141] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038035] Spectre V2 : User space: Vulnerable [ 0.039011] Speculative Store Bypass: Vulnerable [ 0.042578] debug: unmapping init [mem 0xffffffff84459000-0xffffffff84460fff] [ 0.045193] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046727] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047026] ... version: 2 [ 0.048014] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050018] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052015] ... fixed-purpose events: 3 [ 0.053016] ... event mask: 000000070000000f [ 0.054328] rcu: Hierarchical SRCU implementation. [ 0.056509] smp: Bringing up secondary CPUs ... [ 0.057641] x86: Booting SMP configuration: [ 0.058030] .... node #0, CPUs: #1 #2 #3 [ 0.061529] smp: Brought up 1 node, 4 CPUs [ 0.063014] smpboot: Max logical packages: 1 [ 0.064022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144316] node 0 deferred pages initialised in 77ms [ 0.147088] devtmpfs: initialized [ 0.149339] x86/mm: Memory block size: 128MB [ 0.151912] gcov: version magic: 0x41383552 [ 0.155011] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.157069] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158327] pinctrl core: initialized pinctrl subsystem [ 0.159136] [ 0.159429] ************************************************************* [ 0.161010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163011] ** ** [ 0.164006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166012] ** ** [ 0.167008] ** This means that this kernel is built to expose internal ** [ 0.170013] ** IOMMU data structures, which may compromise security on ** [ 0.172012] ** your system. ** [ 0.174013] ** ** [ 0.176020] ** If you see this message and you are not debugging the ** [ 0.178013] ** kernel, report this immediately to your vendor! ** [ 0.180073] ** ** [ 0.181019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182010] ************************************************************* [ 0.184716] NET: Registered protocol family 16 [ 0.185301] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.188071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.191071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.194466] cpuidle: using governor menu [ 0.195994] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.196531] PCI: Using configuration type 1 for base access [ 0.197000] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.207095] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.209017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.213047] cryptd: max_cpu_qlen set to 1000 [ 0.216204] ACPI: Added _OSI(Module Device) [ 0.217009] ACPI: Added _OSI(Processor Device) [ 0.218012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.220013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.225436] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.230366] ACPI: Interpreter enabled [ 0.231046] ACPI: PM: (supports S0 S3 S4 S5) [ 0.232027] ACPI: Using IOAPIC for interrupt routing [ 0.233088] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.235422] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.245333] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.247044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.250032] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.253093] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.258411] acpiphp: Slot [2] registered [ 0.259090] acpiphp: Slot [5] registered [ 0.260067] acpiphp: Slot [6] registered [ 0.261093] acpiphp: Slot [7] registered [ 0.262116] acpiphp: Slot [8] registered [ 0.264117] acpiphp: Slot [9] registered [ 0.265093] acpiphp: Slot [10] registered [ 0.266115] acpiphp: Slot [3] registered [ 0.267088] acpiphp: Slot [4] registered [ 0.269106] acpiphp: Slot [11] registered [ 0.270122] acpiphp: Slot [12] registered [ 0.272111] acpiphp: Slot [13] registered [ 0.273149] acpiphp: Slot [14] registered [ 0.274105] acpiphp: Slot [15] registered [ 0.274943] acpiphp: Slot [16] registered [ 0.276070] acpiphp: Slot [17] registered [ 0.277085] acpiphp: Slot [18] registered [ 0.278032] acpiphp: Slot [19] registered [ 0.278990] acpiphp: Slot [20] registered [ 0.280096] acpiphp: Slot [21] registered [ 0.281060] acpiphp: Slot [22] registered [ 0.281982] acpiphp: Slot [23] registered [ 0.283084] acpiphp: Slot [24] registered [ 0.285125] acpiphp: Slot [25] registered [ 0.286083] acpiphp: Slot [26] registered [ 0.287017] acpiphp: Slot [27] registered [ 0.288114] acpiphp: Slot [28] registered [ 0.290134] acpiphp: Slot [29] registered [ 0.291099] acpiphp: Slot [30] registered [ 0.293059] acpiphp: Slot [31] registered [ 0.294054] PCI host bridge to bus 0000:00 [ 0.295011] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.297022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.299020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.301021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.303018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.305021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.306179] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.309455] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.310946] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.320019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.325051] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.328020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.330019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.333022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.335348] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.338710] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.340029] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.342802] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.348015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.360016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.363014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.366814] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.374013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.383021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.401019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.409330] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.415020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.421014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.436018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.443562] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.448017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.453016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.463015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.474595] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.479012] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.484011] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.497016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.503013] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.507015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.511013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.521014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.528079] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.535018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.539027] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.560016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.569493] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.571270] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.573272] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.575265] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.577279] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.582105] iommu: Default domain type: Passthrough [ 0.583322] SCSI subsystem initialized [ 0.585162] ACPI: bus type USB registered [ 0.586133] usbcore: registered new interface driver usbfs [ 0.588097] usbcore: registered new interface driver hub [ 0.589063] usbcore: registered new device driver usb [ 0.591132] pps_core: LinuxPPS API ver. 1 registered [ 0.592007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.594047] PTP clock support registered [ 0.596074] EDAC MC: Ver: 3.0.0 [ 0.597106] PCI: Using ACPI for IRQ routing [ 0.598754] NetLabel: Initializing [ 0.600011] NetLabel: domain hash size = 128 [ 0.601006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.602073] NetLabel: unlabeled traffic allowed by default [ 0.604141] vgaarb: loaded [ 0.605221] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.606014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.612235] clocksource: Switched to clocksource kvm-clock [ 0.702066] VFS: Disk quotas dquot_6.6.0 [ 0.703543] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.706042] *** VALIDATE ramfs *** [ 0.707278] *** VALIDATE hugetlbfs *** [ 0.708868] pnp: PnP ACPI init [ 0.711371] pnp: PnP ACPI: found 6 devices [ 0.739283] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.742620] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.744890] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.747111] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.749284] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.751535] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.754248] NET: Registered protocol family 2 [ 0.756760] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.761688] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.765089] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.770338] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.773793] TCP: Hash tables configured (established 65536 bind 65536) [ 0.775988] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.778980] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.781432] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.784285] NET: Registered protocol family 1 [ 0.787362] RPC: Registered named UNIX socket transport module. [ 0.789355] RPC: Registered udp transport module. [ 0.790782] RPC: Registered tcp transport module. [ 0.792008] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.793595] NET: Registered protocol family 44 [ 0.794995] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.796195] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.797506] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.799339] PCI: CLS 0 bytes, default 64 [ 0.800841] Unpacking initramfs... [ 2.176669] debug: unmapping init [mem 0xffff97dbbcc54000-0xffff97dbbffbffff] [ 2.180663] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.183094] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.186250] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.649131] Initialise system trusted keyrings [ 2.650844] Key type blacklist registered [ 2.652073] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.660136] zbud: loaded [ 2.663430] *** VALIDATE nfs *** [ 2.664798] *** VALIDATE nfs4 *** [ 2.667327] pstore: using deflate compression [ 2.671234] Platform Keyring initialized [ 2.763840] NET: Registered protocol family 38 [ 2.765682] Key type asymmetric registered [ 2.767032] Asymmetric key parser 'x509' registered [ 2.768473] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.770726] io scheduler mq-deadline registered [ 2.772120] io scheduler kyber registered [ 2.773296] io scheduler bfq registered [ 2.774599] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.776823] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.779792] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.782378] ACPI: Power Button [PWRF] [ 2.786909] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.792608] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.807581] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.817620] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.833250] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.859953] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.886066] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.889782] Non-volatile memory driver v1.3 [ 2.891400] Linux agpgart interface v0.103 [ 2.919929] virtio_blk virtio1: [vda] 149952 512-byte logical blocks (76.8 MB/73.2 MiB) [ 2.923030] vda: detected capacity change from 0 to 76775424 [ 2.939402] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.942292] vdb: detected capacity change from 0 to 1073741824 [ 2.966910] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.969929] vdc: detected capacity change from 0 to 2621440000 [ 2.985262] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.987239] vdd: detected capacity change from 0 to 2621440000 [ 3.002023] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.004192] vde: detected capacity change from 0 to 4294967296 [ 3.018383] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.021059] vdf: detected capacity change from 0 to 4294967296 [ 3.028194] libphy: Fixed MDIO Bus: probed [ 3.041505] usbcore: registered new interface driver usbserial_generic [ 3.044072] usbserial: USB Serial support registered for generic [ 3.045994] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.049971] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.051502] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.053860] mousedev: PS/2 mouse device common for all mice [ 3.056612] rtc_cmos 00:05: RTC can wake from S4 [ 3.059727] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.062817] rtc_cmos 00:05: registered as rtc0 [ 3.064528] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.066816] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.068784] intel_pstate: CPU model not supported [ 3.074767] hid: raw HID events driver (C) Jiri Kosina [ 3.076202] usbcore: registered new interface driver usbhid [ 3.077771] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.078840] usbhid: USB HID core driver [ 3.078986] drop_monitor: Initializing network drop monitor service [ 3.079141] Initializing XFRM netlink socket [ 3.079522] NET: Registered protocol family 10 [ 3.081664] Segment Routing with IPv6 [ 3.090087] NET: Registered protocol family 17 [ 3.091523] mpls_gso: MPLS GSO support [ 3.096944] RAS: Correctable Errors collector initialized. [ 3.099264] AVX version of gcm_enc/dec engaged. [ 3.101272] AES CTR mode by8 optimization enabled [ 3.161066] sched_clock: Marking stable (3161030262, 0)->(4015317250, -854286988) [ 3.163484] registered taskstats version 1 [ 3.164896] Loading compiled-in X.509 certificates [ 3.166660] zswap: loaded using pool lzo/zbud [ 3.189615] Key type big_key registered [ 3.203313] Key type encrypted registered [ 3.205596] ima: No TPM chip found, activating TPM-bypass! [ 3.208119] ima: Allocated hash algorithm: sha1 [ 3.209836] ima: No architecture policies found [ 3.211086] evm: Initialising EVM extended attributes: [ 3.212878] evm: security.selinux [ 3.214385] evm: security.ima [ 3.215592] evm: security.capability [ 3.217151] evm: HMAC attrs: 0x1 [ 3.219822] rtc_cmos 00:05: setting system clock to 2026-09-17 19:29:27 UTC (1789673367) [ 3.226302] debug: unmapping init [mem 0xffffffff85403000-0xffffffff855fffff] [ 3.229431] debug: unmapping init [mem 0xffffffff84182000-0xffffffff84458fff] [ 3.238076] Write protecting the kernel read-only data: 28672k [ 3.241672] debug: unmapping init [mem 0xffffffff82803000-0xffffffff829fffff] [ 3.244518] debug: unmapping init [mem 0xffffffff83114000-0xffffffff831fffff] [ 3.284137] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.293597] systemd[1]: Detected virtualization kvm. [ 3.295707] systemd[1]: Detected architecture x86-64. [ 3.297750] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.325385] systemd[1]: No hostname configured. [ 3.327263] systemd[1]: Set hostname to . [ 3.329148] random: systemd: uninitialized urandom read (16 bytes read) [ 3.331301] systemd[1]: Initializing machine ID from random generator. [ 3.380307] random: ln: uninitialized urandom read (6 bytes read) [ 3.460571] random: systemd: uninitialized urandom read (16 bytes read) [ 3.462913] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.468610] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.475790] systemd[1]: Starting Create Volatile Files and Directories... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.956691] device-mapper: uevent: version 1.0.3 [ 3.958786] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.622655] random: fast init done [ 4.629576] virtio_net virtio0 ens2: renamed from eth0 [ 4.675546] scsi host0: ata_piix [ 4.684661] scsi host1: ata_piix [ 4.686443] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.689068] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.079589] dracut-initqueue[596]: RTNETLINK answers: File exists [ 9.727795] random: crng init done [ 9.729227] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.023473] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.082779] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.304462] SELinux: Disabled at runtime. [ 11.366199] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.375178] systemd[1]: Detected virtualization kvm. [ 11.377036] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.797545] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.800523] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.803650] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.806143] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.811297] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.817771] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.823915] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. [ 11.876218] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.217157] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.495820] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.558597] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.645103] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.660541] EDAC sbridge: Ver: 1.1.2 [ 14.022833] Key type dns_resolver registered [ 14.315063] NFS: Registering the id_resolver key type [ 14.316372] Key type id_resolver registered [ 14.317455] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg132-server login: [ 47.474774] libcfs: loading out-of-tree module taints kernel. [ 47.515138] Key type ._llcrypt registered [ 47.516372] Key type .llcrypt registered [ 47.667389] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 72.479545] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 75.341652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 75.372111] alg: No test for adler32 (adler32-zlib) [ 76.848575] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 77.720958] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 79.537230] Key type lgssc registered [ 81.787017] Lustre: Echo OBD driver; http://www.lustre.org/ [ 85.582030] hrtimer: interrupt took 3452569 ns [ 102.882494] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 151.697444] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 166.281287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 166.346042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 167.720969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 167.764306] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 167.884196] Lustre: lustre-MDT0000: new disk, initializing [ 167.965237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 167.979573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 172.709820] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 189.370895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 189.486353] Lustre: 6519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 189.515444] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 189.520156] Lustre: Skipped 1 previous similar message [ 189.609374] Lustre: lustre-MDT0001: new disk, initializing [ 189.675520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 189.701515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 189.716508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 194.604664] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 199.565192] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 212.669256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 213.223505] Lustre: lustre-OST0000: new disk, initializing [ 213.230600] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 213.235790] Lustre: 8458:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 213.415560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 219.699631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 219.715336] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 219.786931] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 220.825742] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.380477] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 236.569564] Lustre: lustre-OST0001: new disk, initializing [ 236.572366] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 236.581439] Lustre: 9533:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 236.658035] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 243.193463] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 244.775041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 244.796031] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 244.894092] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 255.488581] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 266.041271] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 272.772592] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing check_logdir /tmp/testlogs/ [ 277.198932] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing yml_node [ 281.133686] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 283.693689] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 286.954932] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 289.157991] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 17 15:34:11 EDT 2026 [ 313.045165] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 315.324540] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 332.257673] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 332.259416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 332.277543] Lustre: Skipped 1 previous similar message [ 332.285653] Lustre: Skipped 1 previous similar message [ 337.377929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 337.383447] Lustre: Skipped 1 previous similar message [ 337.922249] Lustre: server umount lustre-MDT0000 complete [ 341.862216] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789673706 with bad export cookie 13709346195445121687 [ 341.865719] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 341.886081] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 342.499522] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.502180] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 342.519544] Lustre: Skipped 3 previous similar messages [ 342.531346] Lustre: Skipped 3 previous similar messages [ 346.535069] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 346.543302] Lustre: Skipped 2 previous similar messages [ 351.713348] LustreError: 8976:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 351.714271] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 351.736081] LustreError: 8976:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 351.749957] Lustre: Skipped 1 previous similar message [ 352.737892] LustreError: 6526:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 352.753985] LustreError: 6526:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 356.320174] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 356.710927] Lustre: server umount lustre-MDT0001 complete [ 361.035926] Lustre: server umount lustre-OST0000 complete [ 365.050771] Lustre: server umount lustre-OST0001 complete [ 372.618990] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 382.660918] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 422.011052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 422.364817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 422.421512] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 422.555437] Lustre: lustre-MDT0000: new disk, initializing [ 422.682513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 422.721867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 427.412898] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 439.661480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 439.777885] Lustre: 15089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 439.864830] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 439.878374] Lustre: Skipped 1 previous similar message [ 440.019099] Lustre: lustre-MDT0001: new disk, initializing [ 440.157800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 440.267481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 440.279394] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 445.614742] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 454.922948] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 462.493100] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 471.190552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 471.648726] Lustre: lustre-OST0000: new disk, initializing [ 471.652546] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 471.667707] Lustre: 17057:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 471.874175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 472.939626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 472.951885] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 473.198246] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 480.454148] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 490.411470] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 494.531111] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 494.695765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 499.412708] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 499.655960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 503.782069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 503.785969] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 503.796961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 503.796969] Lustre: Skipped 2 previous similar messages [ 503.848541] Lustre: Skipped 1 previous similar message [ 507.883244] Lustre: server umount lustre-OST0000 complete [ 514.019631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 514.033980] Lustre: Skipped 1 previous similar message [ 518.960476] Lustre: server umount lustre-MDT0000 complete [ 519.155420] LustreError: 15099:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 519.192726] LustreError: 15099:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 523.206204] LustreError: 15079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789673887 with bad export cookie 13709346195445123220 [ 523.211994] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 523.214230] LustreError: 15079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 523.587254] Lustre: server umount lustre-MDT0001 complete [ 530.569393] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 15:38:13 (1789673893) [ 551.349760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 552.010782] LustreError: 19932:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 552.127538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 557.424614] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 557.559169] LustreError: 19933:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.256921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 572.373265] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 579.830981] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 585.399618] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 587.430635] Lustre: Failing over lustre-MDT0000 [ 587.712641] Lustre: server umount lustre-MDT0000 complete [ 588.260086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 588.278639] LustreError: Skipped 1 previous similar message [ 588.287075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 588.304964] LustreError: 19932:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 588.324660] LustreError: 19932:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 608.613425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 608.736230] Lustre: 3653:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789673957/real 1789673957] req@ffff97db01d85180 x1876608618235904/t0(0) o400->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789673973 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 608.737300] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 608.771752] Lustre: 3653:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 608.789921] LustreError: 19932:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 608.808493] LustreError: 19932:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 620.001035] LustreError: 3649:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97dc0503d500 x1876608618238464/t0(0) o250->MGC192.168.201.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 620.476779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 620.481666] Lustre: Skipped 1 previous similar message [ 620.541562] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 625.404595] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 625.636088] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 625.663486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 625.691388] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 635.459859] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 640.994309] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 641.000732] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 641.018912] Lustre: Skipped 1 previous similar message [ 641.030742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 641.034694] Lustre: Skipped 3 previous similar messages [ 645.698968] Lustre: server umount lustre-MDT0000 complete [ 648.168530] LustreError: 19932:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 648.187051] LustreError: 19932:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 649.956779] LustreError: 19913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789674014 with bad export cookie 13709346195445124634 [ 649.960517] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 649.972212] LustreError: 19913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 650.459287] Lustre: server umount lustre-MDT0001 complete [ 659.404333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 664.652366] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 673.389193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 678.026398] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 685.765595] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 692.162162] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 694.560273] Lustre: Failing over lustre-MDT0000 [ 694.737938] Lustre: server umount lustre-MDT0000 complete [ 695.271486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 695.282307] Lustre: Skipped 1 previous similar message [ 711.648410] Lustre: 3650:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789674059/real 1789674059] req@ffff97dc09c66300 x1876608618266112/t0(0) o400->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789674075 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 711.673473] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 714.749851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 716.769947] LustreError: 23268:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 716.778670] LustreError: 23268:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 720.865328] LustreError: 3649:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97dc0503c000 x1876608618269056/t0(0) o250->MGC192.168.201.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 721.284348] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 725.930146] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 726.503150] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 726.505715] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 726.528265] Lustre: Skipped 1 previous similar message [ 726.551221] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 734.763662] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 738.686240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 738.704716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 738.707411] Lustre: Skipped 2 previous similar messages [ 745.185568] Lustre: server umount lustre-MDT0000 complete [ 750.924363] LustreError: 23254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789674115 with bad export cookie 13709346195445125572 [ 750.927590] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 750.932574] LustreError: 23254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 751.284470] Lustre: server umount lustre-MDT0001 complete [ 762.442247] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 15:42:04 (1789674124) [ 778.486725] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 787.783093] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 829.421706] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 841.414705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 841.686631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 841.725044] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 841.817793] Lustre: lustre-MDT0000: new disk, initializing [ 841.899257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 841.916508] Lustre: Skipped 3 previous similar messages [ 841.933899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 846.923306] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 857.919130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 858.019557] Lustre: 29647:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 858.378996] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 863.289496] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 873.127336] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 879.402725] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 886.353511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 886.727216] Lustre: 31616:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 887.969803] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 888.016235] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 888.122695] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 888.144666] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 893.597752] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 904.996776] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 910.963639] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 934.381987] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 934.389930] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 934.398666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 934.398676] Lustre: Skipped 2 previous similar messages [ 934.441882] Lustre: Skipped 2 previous similar messages [ 945.120169] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 945.727540] Lustre: server umount lustre-OST0000 complete [ 956.407398] Lustre: server umount lustre-MDT0000 complete [ 959.978818] LustreError: 29654:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 960.026230] LustreError: 29654:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 961.321964] LustreError: 31615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789674325 with bad export cookie 13709346195445125978 [ 961.323547] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 961.331959] LustreError: 31615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 961.778734] Lustre: server umount lustre-MDT0001 complete [ 976.494530] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 980.542431] Key type lgssc unregistered [ 981.148944] LNet: 33597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.159093] LNetError: 33597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.182298] LNet: Removed LNI 192.168.201.132@tcp [ 982.558153] Key type .llcrypt unregistered [ 982.561886] Key type ._llcrypt unregistered [ 1011.723325] Key type ._llcrypt registered [ 1011.728704] Key type .llcrypt registered [ 1011.913977] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 1014.069602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1014.105206] alg: No test for adler32 (adler32-zlib) [ 1015.043208] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 1015.328724] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 1017.139540] Key type lgssc registered [ 1018.418957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1029.192570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1029.220166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1031.032710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1036.014674] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1041.379039] LustreError: 35008:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1041.404256] LustreError: 35008:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1044.150995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1044.557248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1049.410479] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1057.712577] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1063.809312] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1072.425749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1073.044483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1076.107865] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1081.440677] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1089.399180] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1089.680632] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1105.285446] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 15:47:47 (1789674467) [ 1107.516876] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1109.530083] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1116.643524] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 15:47:59 (1789674479) [ 1125.600346] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 15:48:08 (1789674488) [ 1126.742777] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1136.374386] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1136.377250] Lustre: Skipped 2 previous similar messages [ 1138.271984] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 15:48:20 (1789674500) [ 1141.286231] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1141.289896] Lustre: Skipped 2 previous similar messages [ 1151.632363] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1151.638176] Lustre: Skipped 4 previous similar messages [ 1168.385542] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 15:48:50 (1789674530) [ 1171.758693] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1175.256282] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1184.388252] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 15:49:06 (1789674546) [ 1186.377405] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1186.384251] Lustre: Skipped 6 previous similar messages [ 1199.961618] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 15:49:22 (1789674562) [ 1214.439767] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1214.455740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1214.469240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1217.508302] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1217.509407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1217.518238] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1217.539833] Lustre: Skipped 1 previous similar message [ 1220.407484] Lustre: server umount lustre-OST0000 complete [ 1227.747680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1227.769034] Lustre: Skipped 1 previous similar message [ 1227.769635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1227.785607] Lustre: Skipped 1 previous similar message [ 1230.216539] Lustre: server umount lustre-MDT0000 complete [ 1232.865359] LustreError: 35008:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1232.893886] LustreError: 35008:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1233.973157] LustreError: 34989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789674598 with bad export cookie 2446488943571573707 [ 1233.978893] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1233.989567] LustreError: 34989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1234.438976] Lustre: server umount lustre-MDT0001 complete [ 1244.902126] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 1247.103502] Key type lgssc unregistered [ 1247.370738] LNet: 42852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1247.375917] LNetError: 42852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1247.389800] LNet: Removed LNI 192.168.201.132@tcp [ 1248.341113] Key type .llcrypt unregistered [ 1248.343451] Key type ._llcrypt unregistered [ 1265.079020] Key type ._llcrypt registered [ 1265.081119] Key type .llcrypt registered [ 1265.217513] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 1279.280152] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 1280.346296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1280.385245] alg: No test for adler32 (adler32-zlib) [ 1281.374962] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 1281.589638] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 1283.248216] Key type lgssc registered [ 1284.304627] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1315.836955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1315.899110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1317.221269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1317.300689] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1317.470187] Lustre: lustre-MDT0000: new disk, initializing [ 1317.593328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1317.618247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1323.505387] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1335.606063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1335.798722] Lustre: 46001:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1335.910978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1335.916826] Lustre: Skipped 1 previous similar message [ 1336.040251] Lustre: lustre-MDT0001: new disk, initializing [ 1336.161062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1336.220537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1336.238734] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1341.061408] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1351.785658] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1358.619903] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1365.528951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1365.770578] Lustre: lustre-OST0000: new disk, initializing [ 1365.773810] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1365.777826] Lustre: 47968:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1365.917588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1372.062698] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1372.758874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1372.769783] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1372.857595] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1382.039818] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1386.818144] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1386.990171] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1391.046052] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1391.251347] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1393.633983] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1393.639449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1393.642802] LustreError: Skipped 1 previous similar message [ 1393.648376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1393.662406] Lustre: Skipped 1 previous similar message [ 1393.677188] Lustre: Skipped 1 previous similar message [ 1398.768155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1398.783152] Lustre: Skipped 1 previous similar message [ 1399.145473] Lustre: server umount lustre-OST0000 complete [ 1403.878858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1403.908387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1403.918351] Lustre: Skipped 1 previous similar message [ 1408.996119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1409.002536] Lustre: Skipped 1 previous similar message [ 1409.509271] Lustre: server umount lustre-MDT0000 complete [ 1413.185796] LustreError: 45994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789674777 with bad export cookie 17155959682375233054 [ 1413.186803] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1413.192050] LustreError: 45994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1413.681776] Lustre: server umount lustre-MDT0001 complete [ 1423.544121] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 15:53:06 (1789674786) [ 1429.590885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1429.964698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1434.440530] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1435.107478] LustreError: 50431:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1435.127748] LustreError: 50431:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1440.228874] LustreError: 50432:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1443.996977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1444.496272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1449.303866] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1457.516613] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1463.904979] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1470.937516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1471.483696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1477.842660] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1486.232474] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1494.201587] LustreError: 52959:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1495.173785] Lustre: Modifying parameter general.timeout=1 in log params [ 1496.301330] Lustre: Modifying parameter general.timeout=2 in log params [ 1498.910637] Lustre: Modifying parameter general.timeout=5 in log params [ 1498.914548] Lustre: Skipped 2 previous similar messages [ 1503.465756] Lustre: Modifying parameter general.timeout=10 in log params [ 1503.468620] Lustre: Skipped 4 previous similar messages [ 1511.742265] Lustre: Modifying parameter general.timeout=19 in log params [ 1511.749376] Lustre: Skipped 8 previous similar messages [ 1528.310454] Lustre: Modifying parameter general.timeout=36 in log params [ 1528.317877] Lustre: Skipped 16 previous similar messages [ 1560.429751] Lustre: Modifying parameter general.timeout=69 in log params [ 1560.434362] Lustre: Skipped 32 previous similar messages [ 1632.843836] Lustre: Disabling parameter general.timeout= in log params [ 1632.847399] Lustre: Skipped 60 previous similar messages [ 1636.907530] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 15:56:39 (1789674999) [ 1640.218382] Lustre: 59922:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1642.727298] Lustre: 60020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1645.104962] Lustre: 60120:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1650.596499] Lustre: 60429:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1650.604733] Lustre: 60429:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1679.273389] Lustre: 61788:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1679.309312] LustreError: 61788:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1683.921257] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 15:57:26 (1789675046) [ 1703.399569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1703.403804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1703.444411] Lustre: Skipped 3 previous similar messages [ 1703.460079] Lustre: Skipped 1 previous similar message [ 1706.944976] Lustre: server umount lustre-MDT0000 complete [ 1708.518823] LustreError: 50432:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1708.554032] LustreError: 50432:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1712.424172] LustreError: 50412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789675076 with bad export cookie 17155959682375233936 [ 1712.441230] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1712.452040] LustreError: 50412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1713.340687] Lustre: server umount lustre-MDT0001 complete [ 1728.672632] Lustre: server umount lustre-OST0000 complete [ 1757.182058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1757.403786] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1757.756873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1761.485611] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1770.214760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1770.437695] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1770.545294] Lustre: 64323:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1777.390462] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1787.265444] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1794.127715] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1801.901739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1802.046873] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1802.196207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1802.198402] Lustre: Skipped 1 previous similar message [ 1807.803883] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1816.062274] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1818.575130] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1818.589711] Lustre: Skipped 23 previous similar messages [ 1818.657618] Lustre: 66887:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1829.859508] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1829.873928] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1829.900464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1829.912484] Lustre: Skipped 1 previous similar message [ 1835.174237] Lustre: server umount lustre-OST0000 complete [ 1842.659511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1842.668386] LustreError: Skipped 1 previous similar message [ 1842.684846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1842.704209] Lustre: Skipped 1 previous similar message [ 1847.469571] Lustre: server umount lustre-MDT0000 complete [ 1850.850248] LustreError: 64333:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1850.867718] LustreError: 64333:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1851.326256] LustreError: 64313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789675215 with bad export cookie 17155959682375237149 [ 1851.332273] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1851.343782] LustreError: 64313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1851.837144] Lustre: server umount lustre-MDT0001 complete [ 1864.162240] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 1867.162210] Key type lgssc unregistered [ 1867.503560] LNet: 68100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1867.538179] LNetError: 68100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1867.566826] LNet: Removed LNI 192.168.201.132@tcp [ 1868.662640] Key type .llcrypt unregistered [ 1868.666716] Key type ._llcrypt unregistered [ 1895.878778] Key type ._llcrypt registered [ 1895.880641] Key type .llcrypt registered [ 1895.958537] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 1897.821593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1897.862465] alg: No test for adler32 (adler32-zlib) [ 1898.970466] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 1899.306474] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 1901.019241] Key type lgssc registered [ 1902.552509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1911.593724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1911.616266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1913.405922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1918.287866] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1923.554313] LustreError: 69526:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1923.575280] LustreError: 69526:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1926.948516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1927.539510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1932.982750] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1942.648675] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1949.479670] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1957.362985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1958.198486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1965.586700] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 1975.766863] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1984.669806] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1998.313679] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1998.331742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1998.343905] Lustre: Skipped 2 previous similar messages [ 1998.371222] Lustre: Skipped 2 previous similar messages [ 2002.436538] Lustre: server umount lustre-MDT0000 complete [ 2003.435322] LustreError: 70697:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2003.472685] LustreError: 70697:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2007.403146] LustreError: 69508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789675371 with bad export cookie 1596343735271410540 [ 2007.412394] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2007.421485] LustreError: 69508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2007.821474] Lustre: server umount lustre-MDT0001 complete [ 2022.153451] Lustre: server umount lustre-OST0000 complete [ 2051.435407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2051.639940] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2051.663589] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2051.988691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2057.164436] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2066.062891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2066.126314] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2066.157696] Lustre: 74400:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2066.691338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2072.102394] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2080.114301] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2086.451471] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2093.124900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2093.435894] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2094.015487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2100.626899] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2108.924490] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2110.916295] Lustre: 76975:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2113.757150] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2121.698156] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2121.709369] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2121.732911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2122.211443] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2125.683929] Lustre: server umount lustre-OST0000 complete [ 2132.451235] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2132.464896] Lustre: Skipped 1 previous similar message [ 2132.469348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2132.481767] Lustre: Skipped 3 previous similar messages [ 2136.657678] Lustre: server umount lustre-MDT0000 complete [ 2137.571031] LustreError: 74411:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2137.592546] LustreError: 74411:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2141.191478] LustreError: 76977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789675505 with bad export cookie 1596343735271411387 [ 2141.197970] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2141.201356] LustreError: 76977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2141.949626] Lustre: server umount lustre-MDT0001 complete [ 2154.934390] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 2157.913576] Key type lgssc unregistered [ 2158.308970] LNet: 78188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2158.316850] LNetError: 78188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2159.339716] LNet: Removed LNI 192.168.201.132@tcp [ 2160.332416] Key type .llcrypt unregistered [ 2160.334345] Key type ._llcrypt unregistered [ 2188.563299] Key type ._llcrypt registered [ 2188.566144] Key type .llcrypt registered [ 2188.706272] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 2190.243302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2190.324416] alg: No test for adler32 (adler32-zlib) [ 2191.382954] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 2191.663441] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 2193.448268] Key type lgssc registered [ 2194.682570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2204.136482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2204.162975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2205.726616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2211.565320] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2214.886000] LustreError: 79599:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2214.909281] LustreError: 79599:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2220.004344] LustreError: 79598:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2221.889422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2222.547877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2228.241654] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2239.011538] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2246.387422] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2255.193669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2256.130984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2263.813596] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2272.940436] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2284.913833] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2285.850773] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2287.596749] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2287.600730] Lustre: Skipped 1 previous similar message [ 2290.647598] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2290.650275] Lustre: Skipped 2 previous similar messages [ 2295.439240] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2295.443558] Lustre: Skipped 4 previous similar messages [ 2303.574780] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2303.578989] Lustre: Skipped 8 previous similar messages [ 2320.265547] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2320.269814] Lustre: Skipped 17 previous similar messages [ 2353.062976] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2353.067849] Lustre: Skipped 31 previous similar messages [ 2392.732723] Lustre: 87729:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2395.480094] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 16:09:18 (1789675758) [ 2396.928677] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2398.365134] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 16:09:21 (1789675761) [ 2403.825417] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2403.829610] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2403.858681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2403.876379] Lustre: Skipped 1 previous similar message [ 2408.298471] Lustre: server umount lustre-OST0000 complete [ 2414.056223] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2414.061347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2414.080874] Lustre: Skipped 2 previous similar messages [ 2418.920101] Lustre: server umount lustre-MDT0000 complete [ 2419.176085] LustreError: 79593:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2419.212691] LustreError: 79593:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2424.291745] LustreError: 79594:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2424.322906] LustreError: 79594:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2424.684685] LustreError: 81453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789675788 with bad export cookie 7827496656642347224 [ 2424.706071] LustreError: 81453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2424.712035] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2425.644843] Lustre: server umount lustre-MDT0001 complete [ 2437.656498] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 2440.381249] Key type lgssc unregistered [ 2440.600939] LNet: 88976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2440.608422] LNetError: 88976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2440.624895] LNet: Removed LNI 192.168.201.132@tcp [ 2441.457938] Key type .llcrypt unregistered [ 2441.462213] Key type ._llcrypt unregistered [ 2445.845874] Key type ._llcrypt registered [ 2445.847876] Key type .llcrypt registered [ 2445.952869] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2451.357241] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules [ 2452.687088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2452.710786] alg: No test for adler32 (adler32-zlib) [ 2453.724143] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 2453.931605] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 2455.680230] Key type lgssc registered [ 2457.143193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2458.452540] LustreError: 90188:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2478.697379] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 2480.334713] LustreError: 90188:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2488.209968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2488.226801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2490.052093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2495.642941] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2500.579632] LustreError: 91202:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2500.604732] LustreError: 91202:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2505.449234] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 16:11:08 (1789675868) [ 2505.698333] LustreError: 91203:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2510.226283] LustreError: 91234:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2510.247239] LustreError: 91234:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2510.263685] LustreError: 91234:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2510.719391] Lustre: server umount lustre-MDT0000 complete [ 2525.005940] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 2527.814921] Key type lgssc unregistered [ 2527.992728] LNet: 92728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2527.996815] LNetError: 92728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2528.007466] LNet: Removed LNI 192.168.201.132@tcp [ 2528.648383] Key type .llcrypt unregistered [ 2528.655349] Key type ._llcrypt unregistered [ 2549.869423] Key type ._llcrypt registered [ 2549.878492] Key type .llcrypt registered [ 2550.240045] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 2552.161149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2552.545875] alg: No test for adler32 (adler32-zlib) [ 2553.739170] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 2554.126667] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 2555.856346] Key type lgssc registered [ 2556.887671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2564.091195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2564.132598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2565.537786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2568.564661] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2574.832180] LustreError: 93947:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2574.864444] LustreError: 93947:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2577.752633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2578.751082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2583.839339] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2592.209700] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2599.568260] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2607.171484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2607.872665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2614.847334] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2622.603532] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2676.340590] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 16:13:59 (1789676039) [ 2678.473784] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2680.305414] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 16:14:02 (1789676042) [ 2685.923837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2685.926314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2685.939140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2690.019222] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2690.022522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2690.045830] Lustre: Skipped 1 previous similar message [ 2690.065821] Lustre: Skipped 2 previous similar messages [ 2692.393819] Lustre: server umount lustre-MDT0000 complete [ 2695.160792] LustreError: 93947:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2695.180246] LustreError: 93947:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2696.467233] LustreError: 93928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789676060 with bad export cookie 14995307350389009705 [ 2696.479392] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2696.883555] Lustre: server umount lustre-MDT0001 complete [ 2710.190447] Lustre: server umount lustre-OST0000 complete [ 2722.980577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2723.378925] LustreError: 98239:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2723.600821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2728.365429] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2728.940653] LustreError: 98240:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2733.026626] LustreError: 98239:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2737.291439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2737.798675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2742.594555] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2750.088634] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2756.284681] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2773.459992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2773.823744] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2773.829516] LustreError: 98229:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2773.859779] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2773.869786] LustreError: 100323:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2773.875801] LustreError: 100323:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2773.879675] LustreError: 100323:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2773.964567] Lustre: server umount lustre-OST0000 complete [ 2773.967042] LustreError: 100323:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2784.239381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2784.364535] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2784.373133] LustreError: 98230:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2784.393249] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2784.400649] LustreError: 100790:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2784.406773] LustreError: 100790:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2784.411523] LustreError: 100790:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2784.471222] Lustre: server umount lustre-OST0000 complete [ 2784.474557] LustreError: 100790:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2793.790199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2793.979885] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2793.982846] Lustre: Found index 0 for lustre-OST0000, updating log [ 2793.985494] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2794.043275] Lustre: lustre-OST0000: new disk, initializing [ 2794.048945] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2794.057623] Lustre: 101326:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2794.282103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2799.887531] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2804.515142] Lustre: Failing over lustre-OST0000 [ 2804.742249] Lustre: server umount lustre-OST0000 complete [ 2812.904779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2812.914779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2812.929096] Lustre: Skipped 1 previous similar message [ 2812.960271] Lustre: Skipped 1 previous similar message [ 2816.112490] Lustre: server umount lustre-MDT0000 complete [ 2818.023608] LustreError: 99927:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2818.065025] LustreError: 99927:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2820.392587] LustreError: 98221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789676184 with bad export cookie 14995307350389012456 [ 2820.394658] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2820.400491] LustreError: 98221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2820.735208] Lustre: server umount lustre-MDT0001 complete [ 2830.453549] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 16:16:32 (1789676192) [ 2849.948264] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 2862.845817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2863.329749] LustreError: 103672:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2863.500240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2869.196845] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2880.085510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2886.991933] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2890.465859] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2897.496290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2898.164229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2898.170890] Lustre: Skipped 1 previous similar message [ 2905.344243] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2915.259905] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2915.334867] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2915.408458] Lustre: lustre-OST0001: new disk, initializing [ 2915.412286] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2915.421915] Lustre: 105992:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2922.576719] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2925.274472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2925.281915] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2925.418285] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2930.833558] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2938.744913] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2946.022711] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2946.026124] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2946.044651] Lustre: Skipped 2 previous similar messages [ 2946.074176] Lustre: Skipped 2 previous similar messages [ 2949.238212] Lustre: server umount lustre-MDT0001 complete [ 2951.144414] LustreError: 103673:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2951.182976] LustreError: 103673:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2960.075135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2960.237935] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2960.248141] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2960.361785] Lustre: 103661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2962.129269] LustreError: 107561:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2962.146678] LustreError: 107561:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2962.155829] LustreError: 103664:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2962.305864] LustreError: 107572:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.132@tcp: cfg command failed: rc = -17 [ 2962.322624] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2962.322624] [ 2965.673277] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 2966.521169] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2977.520857] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 16:18:59 (1789676339) [ 3002.118103] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 3022.584634] Lustre: 109628:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3045.875456] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3050.294342] Lustre: 110763:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3134.955339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3134.968786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3134.982432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3141.169891] Lustre: server umount lustre-MDT0000 complete [ 3144.940789] LustreError: 106710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789676509 with bad export cookie 14995307350389013450 [ 3144.945162] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3144.947982] LustreError: 106710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3145.297247] Lustre: server umount lustre-MDT0001 complete [ 3159.933119] Lustre: server umount lustre-OST0000 complete [ 3161.696232] Lustre: 93258:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789676509/real 1789676509] req@ffff97db0cbf1500 x1876611215755648/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789676525 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3161.715141] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3161.738457] Lustre: Skipped 3 previous similar messages [ 3164.502033] Lustre: server umount lustre-OST0001 complete [ 3171.414608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3182.389510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3191.607041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3201.981792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3226.096959] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 3239.643256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3239.670057] Lustre: lustre-MDT0000: reset Object Index mappings [ 3240.230275] LustreError: 114717:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3240.242421] LustreError: 114717:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 3240.348078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3240.358031] Lustre: Skipped 2 previous similar messages [ 3245.745132] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 3255.906223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3255.944286] Lustre: lustre-MDT0001: reset Object Index mappings [ 3262.054576] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 3265.427606] Lustre: 115866:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3272.937468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3280.767844] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3281.328616] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 3284.878055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3291.221944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3297.292140] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3297.305657] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3298.706824] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 3307.369571] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3310.654805] Lustre: 117742:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3362.283498] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 16:25:24 (1789676724) [ 3389.410370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3389.418520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3389.424668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3389.424677] Lustre: Skipped 8 previous similar messages [ 3389.453732] Lustre: Skipped 2 previous similar messages [ 3394.533155] LustreError: 114714:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3394.550391] LustreError: 114714:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 3394.586315] Lustre: server umount lustre-MDT0000 complete [ 3397.917870] LustreError: 114697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789676762 with bad export cookie 14995307350389060322 [ 3397.922481] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3397.932089] LustreError: 114697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3404.402763] Lustre: server umount lustre-MDT0001 complete [ 3414.596615] Lustre: server umount lustre-OST0000 complete [ 3419.067273] LustreError: 118705:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000400:0xb5:0x0].0x0 (ffff97db0d04c200) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3425.450858] Lustre: server umount lustre-OST0001 complete [ 3432.858873] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 3443.261537] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 3483.902110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3484.236467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3484.286337] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3484.368706] Lustre: lustre-MDT0000: new disk, initializing [ 3484.488214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3484.494676] Lustre: Skipped 3 previous similar messages [ 3484.523711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3489.868561] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 3498.013975] Lustre: server umount lustre-MDT0000 complete [ 3509.002826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3509.290395] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3509.305882] Lustre: 122013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3509.319877] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3514.907641] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 3520.405954] Lustre: server umount lustre-MDT0000 complete [ 3530.697966] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 16:28:12 (1789676892) [ 3532.569669] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3534.843275] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 16:28:17 (1789676897) [ 3556.974490] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 3568.321877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3572.968984] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 3584.822319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3584.906854] Lustre: 124066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3585.033060] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3585.050370] Lustre: Skipped 1 previous similar message [ 3585.227955] Lustre: lustre-MDT0001: new disk, initializing [ 3585.542281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3585.550738] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3592.259462] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 3598.300733] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3605.796814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3606.013641] Lustre: lustre-OST0000: new disk, initializing [ 3606.015946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3606.021888] Lustre: 125644:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3607.853430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3607.864771] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3607.997476] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3612.686078] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 3625.522155] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3625.788674] Lustre: lustre-OST0001: new disk, initializing [ 3625.797230] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3625.806247] Lustre: 126515:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3627.678099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3627.693988] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3627.841097] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3633.637471] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 3644.938966] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3649.271632] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4054.344454] Lustre: 132940:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4054.352949] Lustre: 132940:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4054.359894] Lustre: 132940:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4054.365990] Lustre: 132940:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4054.374829] Lustre: 132940:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4054.379487] Lustre: 132940:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4372.407973] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:42:14 (1789677734) [ 4389.860475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4389.867798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4389.889278] Lustre: Skipped 3 previous similar messages [ 4389.892606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4389.896185] Lustre: Skipped 7 previous similar messages [ 4394.346975] Lustre: server umount lustre-MDT0000 complete [ 4396.012294] LustreError: 127333:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4396.036269] LustreError: 127333:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4398.462558] LustreError: 124058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789677762 with bad export cookie 14995307350389127956 [ 4398.471745] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4398.475879] LustreError: 124058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4401.123691] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4401.139651] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4401.145502] Lustre: Skipped 4 previous similar messages [ 4401.160794] Lustre: Skipped 4 previous similar messages [ 4405.361249] Lustre: server umount lustre-MDT0001 complete [ 4417.538255] Lustre: server umount lustre-OST0000 complete [ 4428.210678] Lustre: server umount lustre-OST0001 complete [ 4435.416592] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 4448.184778] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 4489.770171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4490.061348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4490.101897] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4490.177296] Lustre: lustre-MDT0000: new disk, initializing [ 4490.251267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4490.254276] Lustre: Skipped 5 previous similar messages [ 4490.270753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4496.409153] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 4508.174736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4508.338303] Lustre: 135997:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4508.408959] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4508.414323] Lustre: Skipped 1 previous similar message [ 4508.520416] Lustre: lustre-MDT0001: new disk, initializing [ 4508.744145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4508.759638] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4513.425977] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 4522.824675] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4529.057094] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4536.615658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4536.879496] Lustre: lustre-OST0000: new disk, initializing [ 4536.887016] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4536.894516] Lustre: 137963:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4537.035898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4537.038029] Lustre: Skipped 1 previous similar message [ 4538.804053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4538.818635] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4538.963870] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4544.167299] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 4554.871166] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4565.997681] Lustre: lustre-MDD0000: changelog on [ 4569.137676] Lustre: lustre-MDD0001: changelog on [ 6595.473691] Lustre: lustre-MDD0001: changelog off [ 6597.648654] Lustre: 140131:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6598.718902] Lustre: lustre-MDD0000: changelog off [ 6607.332059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6607.333444] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6607.335826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6607.335834] Lustre: Skipped 3 previous similar messages [ 6607.352127] Lustre: Skipped 1 previous similar message [ 6607.355569] LustreError: Skipped 1 previous similar message [ 6611.381451] Lustre: server umount lustre-OST0000 complete [ 6617.570532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6617.588689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6617.591492] Lustre: Skipped 1 previous similar message [ 6621.198679] Lustre: server umount lustre-MDT0000 complete [ 6622.700274] LustreError: 139773:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6622.732958] LustreError: 139773:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6625.020570] LustreError: 135987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789679989 with bad export cookie 14995307350389950974 [ 6625.028883] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6625.041904] LustreError: 135987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6625.256964] Lustre: server umount lustre-MDT0001 complete [ 6638.005923] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 6641.163307] Key type lgssc unregistered [ 6641.534984] LNet: 147134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6641.542723] LNetError: 147134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6641.557937] LNet: Removed LNI 192.168.201.132@tcp [ 6642.810309] Key type .llcrypt unregistered [ 6642.819886] Key type ._llcrypt unregistered [ 6666.695474] Key type ._llcrypt registered [ 6666.697101] Key type .llcrypt registered [ 6666.891013] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 6687.413425] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 6688.719867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6688.727134] alg: No test for adler32 (adler32-zlib) [ 6690.008175] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 6690.433790] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 6692.161220] Key type lgssc registered [ 6693.470285] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6739.211575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6739.270699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6740.603510] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6740.632857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6740.721825] Lustre: lustre-MDT0000: new disk, initializing [ 6740.814129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6740.830660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6745.769497] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 6757.939934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6758.072225] Lustre: 150334:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6758.124882] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6758.129412] Lustre: Skipped 1 previous similar message [ 6758.207749] Lustre: lustre-MDT0001: new disk, initializing [ 6758.312087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6758.387355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6758.400861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6762.585905] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 6773.571976] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6780.484912] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6788.936814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6789.274474] Lustre: lustre-OST0000: new disk, initializing [ 6789.277395] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6789.280520] Lustre: 152305:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6789.348347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6795.893351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6795.920188] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6796.096520] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6797.101702] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 6807.884715] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6815.151554] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_module obdecho/obdecho [ 6817.510498] LustreError: 153310:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6817.522543] LustreError: 153310:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6819.638705] LustreError: 153406:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6835.169101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6835.176352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6835.189899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6837.229399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6837.240181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6837.273927] Lustre: Skipped 1 previous similar message [ 6838.974349] Lustre: server umount lustre-MDT0000 complete [ 6843.365317] LustreError: 150328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789680207 with bad export cookie 5046010570285863910 [ 6843.376046] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6843.377894] LustreError: 150328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6843.905739] Lustre: server umount lustre-MDT0001 complete [ 6859.451410] Lustre: server umount lustre-OST0000 complete [ 6884.425031] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 6901.582982] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 6948.345259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6948.524039] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6948.547333] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6948.598165] Lustre: lustre-MDT0000: new disk, initializing [ 6948.713621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6948.782638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6955.061960] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 6968.662776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6968.836807] Lustre: 157149:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6968.901494] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6968.905056] Lustre: Skipped 1 previous similar message [ 6968.977930] Lustre: lustre-MDT0001: new disk, initializing [ 6969.084959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6969.134284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6969.154957] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6973.815663] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 6986.221605] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6992.623351] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7001.675186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7002.144928] Lustre: lustre-OST0000: new disk, initializing [ 7002.150689] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7002.179469] Lustre: 159117:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7002.321955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7010.372542] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7011.934519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7011.944251] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7012.099369] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7023.768492] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7030.302251] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7030.634431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7036.297972] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7036.631843] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7043.045741] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7043.050747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7043.058693] Lustre: Skipped 1 previous similar message [ 7043.060742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7043.082803] Lustre: Skipped 3 previous similar messages [ 7048.164565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7048.173987] Lustre: Skipped 1 previous similar message [ 7053.280131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7053.286270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7053.301907] Lustre: Skipped 1 previous similar message [ 7053.540943] Lustre: server umount lustre-OST0000 complete [ 7058.401904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7063.525155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7063.537771] Lustre: Skipped 3 previous similar messages [ 7064.908073] Lustre: server umount lustre-MDT0000 complete [ 7068.643815] LustreError: 157155:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7068.675330] LustreError: 157155:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 7069.744260] LustreError: 157140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789680434 with bad export cookie 5046010570285864869 [ 7069.758969] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7070.246970] Lustre: server umount lustre-MDT0001 complete [ 7074.692629] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 17:27:16 (1789680436) [ 7077.006707] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7079.530763] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 17:27:21 (1789680441) [ 7088.432441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7088.875876] LustreError: 161531:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7088.945466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7094.241694] LustreError: 161532:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7094.722200] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7099.376018] LustreError: 161531:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7104.483128] LustreError: 161532:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7104.839328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7111.028039] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7119.552287] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7126.686656] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7135.674118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7136.147601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7136.161648] Lustre: Skipped 1 previous similar message [ 7144.569704] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7153.646299] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7171.046047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7171.052872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7171.071856] Lustre: Skipped 2 previous similar messages [ 7171.091131] Lustre: Skipped 2 previous similar messages [ 7176.544190] Lustre: server umount lustre-MDT0000 complete [ 7179.784628] LustreError: 163789:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7181.087928] LustreError: 161512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789680545 with bad export cookie 5046010570285865751 [ 7181.106976] LustreError: 161512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7181.112739] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7181.293954] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7181.326781] Lustre: Skipped 1 previous similar message [ 7187.814481] Lustre: server umount lustre-MDT0001 complete [ 7196.242367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7213.622296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7213.842822] LustreError: 165402:0:(llog.c:1655:llog_backup()) MGC192.168.201.132@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7213.847142] Lustre: 165402:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.132@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7215.672886] LustreError: 165407:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7215.681253] LustreError: 165407:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7250.913956] LustreError: 165408:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7250.947421] LustreError: 165408:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 24 previous similar messages [ 7274.467193] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7274.481156] Lustre: 165402:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.132@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7315.436364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7315.442096] LustreError: 165437:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7315.495818] LustreError: 165437:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 45 previous similar messages [ 7320.556733] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7320.911292] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7330.682057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7336.933018] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7336.943228] LustreError: Skipped 1 previous similar message [ 7351.267050] LustreError: 148554:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003bdfd8f2 x1876615552449664/t0(0) o250->MGC192.168.201.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7356.271899] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7356.924396] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7364.441236] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7366.395321] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7372.306970] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7374.183765] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7396.312800] Lustre: server umount lustre-MDT0000 complete [ 7397.871049] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7397.899547] Lustre: Skipped 1 previous similar message [ 7400.936267] LustreError: 161513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789680765 with bad export cookie 5046010570285872240 [ 7400.942782] LustreError: 161513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7400.945857] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7401.270534] Lustre: server umount lustre-MDT0001 complete [ 7416.191750] Lustre: server umount lustre-OST0000 complete [ 7441.070208] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 7452.240245] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 7492.667185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7492.978483] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7493.012152] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7493.122310] Lustre: lustre-MDT0000: new disk, initializing [ 7493.222602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7493.226789] Lustre: Skipped 1 previous similar message [ 7493.252264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7499.090673] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7512.577516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7512.738673] Lustre: 170505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7512.969094] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7517.974728] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7529.258845] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7536.224849] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7544.459274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7544.736263] Lustre: lustre-OST0000: new disk, initializing [ 7544.738180] Lustre: Skipped 1 previous similar message [ 7544.745518] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7544.752615] Lustre: Skipped 2 previous similar messages [ 7544.758503] Lustre: 172476:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7553.053619] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7554.108977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7554.121737] Lustre: Skipped 1 previous similar message [ 7554.128727] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7554.321119] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7563.046492] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7568.283623] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7568.582675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7573.913506] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7574.143941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7580.134197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7580.139133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7580.142656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7580.153069] Lustre: Skipped 2 previous similar messages [ 7580.166658] Lustre: Skipped 8 previous similar messages [ 7582.412696] Lustre: server umount lustre-OST0000 complete [ 7589.857091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7589.864775] LustreError: Skipped 1 previous similar message [ 7592.484673] Lustre: server umount lustre-MDT0000 complete [ 7595.495047] LustreError: 170518:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7595.519662] LustreError: 170518:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 35 previous similar messages [ 7595.869627] LustreError: 170500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789680960 with bad export cookie 5046010570285889040 [ 7595.885172] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7596.328508] Lustre: server umount lustre-MDT0001 complete [ 7604.823277] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 17:36:07 (1789680967) [ 7611.366520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7615.882944] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7623.975679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7629.007183] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7637.256432] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7643.761364] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7651.109812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7658.762680] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7667.086623] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7680.751862] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 17:37:23 (1789681043) [ 7684.577628] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7684.587111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7684.600079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7684.618396] Lustre: Skipped 3 previous similar messages [ 7684.625412] Lustre: Skipped 3 previous similar messages [ 7690.183461] Lustre: server umount lustre-OST0000 complete [ 7700.342908] Lustre: server umount lustre-MDT0000 complete [ 7703.988432] LustreError: 174871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789681068 with bad export cookie 5046010570285889929 [ 7703.991894] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7703.998110] LustreError: 174871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7704.303508] Lustre: server umount lustre-MDT0001 complete [ 7714.874474] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 7717.452253] Key type lgssc unregistered [ 7717.754241] LNet: 178615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7717.773492] LNetError: 178615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7717.793165] LNet: Removed LNI 192.168.201.132@tcp [ 7718.739637] Key type .llcrypt unregistered [ 7718.741145] Key type ._llcrypt unregistered [ 7739.186121] Key type ._llcrypt registered [ 7739.187589] Key type .llcrypt registered [ 7739.310717] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 7740.492229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7740.641868] alg: No test for adler32 (adler32-zlib) [ 7741.648949] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 7741.913790] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 7743.576310] Key type lgssc registered [ 7744.542098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7751.624676] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7751.653695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7767.138207] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7772.257533] LustreError: 179853:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.132@tcp: failed processing log, type 4: rc = -110 [ 7777.376834] LustreError: 179853:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7777.390498] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7777.401498] LustreError: MGC192.168.201.132@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7777.414456] LustreError: 179853:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7777.422358] LustreError: 179853:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 7777.429497] LustreError: 179853:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 7777.526834] Lustre: server umount lustre-OST0000 complete [ 7777.530315] LustreError: 179853:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7787.919223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7789.453832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7794.576508] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7798.762443] LustreError: 180355:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7798.790852] LustreError: 180355:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7803.197744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7803.707435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7808.683282] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7816.612384] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7822.484986] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7828.932783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7829.174187] LustreError: 182174:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7829.184781] Lustre: 182174:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.132@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7829.369481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7835.657543] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7843.457041] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7848.738611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7848.756427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7851.910251] Lustre: server umount lustre-MDT0000 complete [ 7854.055889] LustreError: 181084:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7854.074087] LustreError: 181084:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7855.159439] LustreError: 180336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789681219 with bad export cookie 285601207245401454 [ 7855.168495] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7855.174579] LustreError: 180336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7857.056315] LustreError: 181086:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000afa13946 x1876616655433984/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7857.215740] Lustre: server umount lustre-MDT0001 complete [ 7871.450805] Lustre: server umount lustre-OST0000 complete [ 7880.823913] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 7891.742527] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 7926.600053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7926.930866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7926.983638] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7927.094946] Lustre: lustre-MDT0000: new disk, initializing [ 7927.247263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7927.263179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7930.877635] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7939.939352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7940.071403] Lustre: 185697:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7940.116298] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7940.119691] Lustre: Skipped 1 previous similar message [ 7940.184086] Lustre: lustre-MDT0001: new disk, initializing [ 7940.240784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7940.286621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7940.298625] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7944.081506] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7952.659647] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7957.422371] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7963.848226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7964.090756] Lustre: lustre-OST0000: new disk, initializing [ 7964.094486] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7964.101123] Lustre: 187661:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7964.197586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7970.635405] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 7973.502218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7973.505640] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7973.639716] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7980.238856] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7985.845374] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7986.034902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7991.576456] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7991.752581] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7994.342758] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7994.343461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7994.351498] LustreError: Skipped 1 previous similar message [ 7994.363458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7994.371545] Lustre: Skipped 3 previous similar messages [ 7994.391120] Lustre: Skipped 3 previous similar messages [ 7999.475612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7999.485660] Lustre: Skipped 1 previous similar message [ 7999.867817] Lustre: server umount lustre-OST0000 complete [ 8004.578549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8004.586532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8004.607944] Lustre: Skipped 1 previous similar message [ 8004.614905] Lustre: Skipped 1 previous similar message [ 8009.706702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8009.713467] Lustre: Skipped 1 previous similar message [ 8010.090425] Lustre: server umount lustre-MDT0000 complete [ 8013.464653] LustreError: 185687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789681377 with bad export cookie 285601207245402175 [ 8013.467847] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8013.478425] LustreError: 185687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8013.886848] Lustre: server umount lustre-MDT0001 complete [ 8023.327448] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 17:43:05 (1789681385) [ 8040.222422] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 8043.064208] Key type lgssc unregistered [ 8043.430240] LNet: 190605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8043.433740] LNetError: 190605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8043.448806] LNet: Removed LNI 192.168.201.132@tcp [ 8044.167759] Key type .llcrypt unregistered [ 8044.169831] Key type ._llcrypt unregistered [ 8061.758750] Key type ._llcrypt registered [ 8061.760885] Key type .llcrypt registered [ 8061.859824] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 8062.868100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8062.910757] alg: No test for adler32 (adler32-zlib) [ 8064.079098] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 8064.294233] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 8065.960332] Key type lgssc registered [ 8067.146715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8074.512142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8074.536284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8075.887862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8081.093582] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8086.497620] LustreError: 191824:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8086.515581] LustreError: 191824:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8090.248902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8090.543427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8095.356240] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8103.123824] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8108.439272] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8114.982238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8115.108956] LustreError: 193693:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8115.124552] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8115.126489] LustreError: 193688:0:(llog.c:1696:llog_backup()) MGC192.168.201.132@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8115.131326] Lustre: 193688:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.132@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8115.262013] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8120.919800] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8128.929059] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8131.555881] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8131.558524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8131.568855] Lustre: Skipped 1 previous similar message [ 8131.577389] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8136.675329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8136.683042] Lustre: Skipped 4 previous similar messages [ 8141.794879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8141.798847] Lustre: Skipped 2 previous similar messages [ 8145.376153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8145.657517] Lustre: server umount lustre-MDT0000 complete [ 8146.917915] LustreError: 191824:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8146.935859] LustreError: 191824:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8149.283201] LustreError: 191803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789681513 with bad export cookie 4463704806943440349 [ 8149.285137] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8149.290413] LustreError: 191803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8149.584585] Lustre: server umount lustre-MDT0001 complete [ 8163.598442] Lustre: server umount lustre-OST0000 complete [ 8172.398531] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 8182.801563] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 8216.535882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8216.754539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8216.775503] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8216.826787] Lustre: lustre-MDT0000: new disk, initializing [ 8216.917037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8216.927162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8220.840717] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8229.488194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8229.582694] Lustre: 197216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8229.619372] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8229.622774] Lustre: Skipped 1 previous similar message [ 8229.625868] LustreError: 197998:0:(llog.c:1696:llog_backup()) MGC192.168.201.132@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8229.633420] LustreError: 197998:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8229.638311] Lustre: 197998:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.132@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8229.653803] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8229.658202] Lustre: Skipped 1 previous similar message [ 8229.717287] Lustre: lustre-MDT0001: new disk, initializing [ 8229.806793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8229.854886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8229.869962] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8233.387496] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8241.526707] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8246.402762] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8252.386905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8252.539074] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8252.541129] Lustre: Skipped 1 previous similar message [ 8252.543634] LustreError: 199157:0:(llog.c:1696:llog_backup()) MGC192.168.201.132@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8252.550035] LustreError: 199157:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8252.553130] Lustre: 199157:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.132@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8252.559380] Lustre: 199157:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8252.579363] Lustre: lustre-OST0000: new disk, initializing [ 8252.584984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8252.589732] Lustre: 199182:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8252.657980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8258.432855] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8260.644607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8260.670462] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8260.765166] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8267.103790] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8271.210775] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8271.426333] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8276.207753] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8276.426054] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8281.577529] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8281.581405] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8281.582388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8281.593332] Lustre: Skipped 2 previous similar messages [ 8281.625627] LustreError: Skipped 1 previous similar message [ 8284.727926] Lustre: server umount lustre-OST0000 complete [ 8291.297926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8291.305703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8291.316660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8291.327179] Lustre: Skipped 1 previous similar message [ 8294.579527] Lustre: server umount lustre-MDT0000 complete [ 8296.938989] LustreError: 197226:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8296.963101] LustreError: 197226:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8297.862920] LustreError: 197207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789681662 with bad export cookie 4463704806943441077 [ 8297.873768] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8297.876736] LustreError: 197207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8298.148601] Lustre: server umount lustre-MDT0001 complete [ 8306.132979] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 17:47:48 (1789681668) [ 8320.243818] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 8329.644958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8330.017502] LustreError: 202304:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8330.063940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8334.560588] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8340.454566] LustreError: 202305:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8340.466983] LustreError: 202305:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8342.989277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8347.236564] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8349.981717] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8356.233347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8361.575956] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8369.171641] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8369.336840] Lustre: lustre-OST0001: new disk, initializing [ 8369.339879] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8369.342971] Lustre: 204612:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8369.415864] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8369.419612] Lustre: Skipped 2 previous similar messages [ 8374.793310] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8376.892461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8376.902976] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8376.957082] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8386.038833] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8391.547167] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8393.516939] Lustre: DEBUG MARKER: ADD OST3 [ 8398.445622] loop: module loaded [ 8398.522476] loop0: detected capacity change from 0 to 204800000 [ 8398.806308] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8398.814700] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8398.825101] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8404.203343] loop0: detected capacity change from 0 to 204800000 [ 8405.763775] Lustre: DEBUG MARKER: STOP OST3 [ 8409.835579] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8410.034772] Lustre: lustre-OST0003: new disk, initializing [ 8410.038066] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8410.042219] Lustre: 206592:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8411.440018] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8411.548653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8411.556847] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8411.597226] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8413.473728] LustreError: 206773:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8413.788818] Lustre: Failing over lustre-OST0003 [ 8414.125868] Lustre: server umount lustre-OST0003 complete [ 8415.785059] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8416.743400] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8416.750071] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8416.754468] Lustre: Skipped 1 previous similar message [ 8416.760200] LustreError: 203788:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8424.487136] Lustre: DEBUG MARKER: START OST3 again [ 8430.732266] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8431.011539] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8432.139302] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8432.721849] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8432.723355] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8436.727380] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8456.141395] Lustre: Failing over lustre-OST0003 [ 8456.221176] Lustre: server umount lustre-OST0003 complete [ 8456.680841] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8456.686987] LustreError: 203791:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8456.688695] Lustre: Skipped 2 previous similar messages [ 8456.698505] LustreError: 203791:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 8476.651210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8476.658170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8476.671685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8476.679708] Lustre: Skipped 1 previous similar message [ 8478.943872] Lustre: server umount lustre-MDT0000 complete [ 8481.952811] LustreError: 202285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789681846 with bad export cookie 4463704806943441952 [ 8481.963405] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8481.970645] LustreError: 202285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8488.310336] Lustre: server umount lustre-MDT0001 complete [ 8491.847942] Lustre: server umount lustre-OST0000 complete [ 8495.217962] Lustre: server umount lustre-OST0001 complete [ 8500.467144] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 8507.813766] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 8538.444528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8538.603937] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8538.622722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8538.724971] Lustre: lustre-MDT0000: new disk, initializing [ 8538.778751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8538.781866] Lustre: Skipped 2 previous similar messages [ 8538.791312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8542.099949] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8551.241725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8551.357148] Lustre: 212068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8551.591199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8554.990537] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8563.200950] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8567.494096] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8572.681377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8572.909315] Lustre: lustre-OST0000: new disk, initializing [ 8572.912691] Lustre: Skipped 1 previous similar message [ 8572.917780] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8572.924204] Lustre: Skipped 2 previous similar messages [ 8572.930997] Lustre: 214035:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8574.468971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8574.476093] Lustre: Skipped 1 previous similar message [ 8574.481155] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8574.565832] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8578.547718] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8587.609460] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8591.953587] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8592.156925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8595.976251] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8596.226841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8600.037928] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8600.038749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8600.053118] Lustre: Skipped 5 previous similar messages [ 8600.054614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8600.054620] Lustre: Skipped 8 previous similar messages [ 8603.888455] Lustre: server umount lustre-OST0000 complete [ 8613.368497] Lustre: server umount lustre-MDT0000 complete [ 8615.394578] LustreError: 212081:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8615.414547] LustreError: 212081:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 8616.493524] LustreError: 212872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789681980 with bad export cookie 4463704806943444255 [ 8616.503821] LustreError: 212872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8616.504462] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8616.803157] Lustre: server umount lustre-MDT0001 complete [ 8619.996325] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 17:53:02 (1789681982) [ 8633.043176] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 8641.015243] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 8666.795432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8666.921304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8666.934987] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8666.986289] Lustre: lustre-MDT0000: new disk, initializing [ 8667.035181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8667.037503] Lustre: Skipped 2 previous similar messages [ 8667.044544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8669.841965] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8677.626976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8677.711213] Lustre: 218641:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8677.885446] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8680.922615] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8687.333579] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8690.966664] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8695.653105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8695.819797] Lustre: 220611:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8697.321797] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8701.132919] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8708.062439] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8711.291465] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8711.431397] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8714.706602] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8714.829484] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8717.793177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8717.795664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8717.802707] LustreError: Skipped 2 previous similar messages [ 8717.812461] Lustre: Skipped 3 previous similar messages [ 8717.813795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8717.832509] Lustre: Skipped 5 previous similar messages [ 8722.480436] Lustre: server umount lustre-OST0000 complete [ 8731.525701] Lustre: server umount lustre-MDT0000 complete [ 8733.825453] LustreError: 219437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789682098 with bad export cookie 4463704806943445130 [ 8733.829323] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8733.830580] LustreError: 219437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8734.031411] Lustre: server umount lustre-MDT0001 complete [ 8739.473430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8742.998929] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8745.447145] LustreError: 222827:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8745.464817] LustreError: 222827:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8748.428110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8751.415534] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8756.365636] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8759.842022] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8764.231246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8768.382681] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8773.158976] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8783.844923] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8783.848536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8783.849948] Lustre: Skipped 4 previous similar messages [ 8783.851779] Lustre: Skipped 2 previous similar messages [ 8783.855072] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8783.857462] LustreError: Skipped 1 previous similar message [ 8787.382602] Lustre: server umount lustre-OST0000 complete [ 8796.401352] Lustre: server umount lustre-MDT0000 complete [ 8798.850944] LustreError: 222806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789682163 with bad export cookie 4463704806943446019 [ 8798.853301] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8798.859584] LustreError: 222806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8799.227976] Lustre: server umount lustre-MDT0001 complete [ 8910.012124] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 8911.812633] Key type lgssc unregistered [ 8912.003575] LNet: 226488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8912.009541] LNetError: 226488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8912.019834] LNet: Removed LNI 192.168.201.132@tcp [ 8912.550265] Key type .llcrypt unregistered [ 8912.551926] Key type ._llcrypt unregistered [ 8929.474335] Key type ._llcrypt registered [ 8929.475583] Key type .llcrypt registered [ 8929.532793] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 8939.447899] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 8940.159539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8940.261531] alg: No test for adler32 (adler32-zlib) [ 8941.251849] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 8941.414747] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 8943.050554] Key type lgssc registered [ 8943.904234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8966.275208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8966.298045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8967.446315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8967.465203] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8967.508687] Lustre: lustre-MDT0000: new disk, initializing [ 8967.546794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8967.559474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8970.565321] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8977.306709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8977.353349] Lustre: 229812:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8977.384541] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8977.387217] Lustre: Skipped 1 previous similar message [ 8977.424762] Lustre: lustre-MDT0001: new disk, initializing [ 8977.465814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8977.481420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8977.485079] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8979.423562] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 8985.772744] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8990.307698] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8994.849436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8995.044453] Lustre: lustre-OST0000: new disk, initializing [ 8995.047714] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8995.051273] Lustre: 231777:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8995.110198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8999.366954] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9003.548212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9003.553885] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9003.616948] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9006.101729] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9009.373477] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9009.548774] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9012.792437] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9012.919288] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9018.851992] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9018.852542] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9018.859574] LustreError: Skipped 1 previous similar message [ 9018.860499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9018.865533] Lustre: Skipped 1 previous similar message [ 9020.267895] Lustre: server umount lustre-OST0000 complete [ 9023.460730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9023.467227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9023.481218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9023.488046] Lustre: Skipped 1 previous similar message [ 9028.477956] Lustre: server umount lustre-MDT0000 complete [ 9030.696047] LustreError: 229802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789682394 with bad export cookie 14973775701575067620 [ 9030.699451] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9030.708693] LustreError: 229802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9030.894163] Lustre: server umount lustre-MDT0001 complete [ 9041.362801] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 9047.625472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9047.816735] LustreError: 234707:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9047.829179] LustreError: 234707:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9047.856459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9050.347942] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9053.155940] LustreError: 234708:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9055.451382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9055.668395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9058.677107] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9060.775271] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9065.279945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9065.502229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9069.746212] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9075.498122] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9075.655677] Lustre: lustre-OST0001: new disk, initializing [ 9075.658752] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9075.666575] Lustre: 237015:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9076.521303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9076.525301] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9076.619727] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9079.623485] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9088.394649] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9092.928198] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9101.929175] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 18:01:05 (1789682465) [ 9106.408989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9106.417418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9106.422512] Lustre: Skipped 4 previous similar messages [ 9106.429558] Lustre: Skipped 4 previous similar messages [ 9111.536080] LustreError: 234708:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9111.551289] LustreError: 234708:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9111.576965] Lustre: server umount lustre-MDT0000 complete [ 9113.904396] LustreError: 237014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789682478 with bad export cookie 14973775701575068509 [ 9113.910318] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9113.914389] LustreError: 237014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9114.115024] Lustre: server umount lustre-MDT0001 complete [ 9125.837246] Lustre: server umount lustre-OST0000 complete [ 9138.519444] Lustre: server umount lustre-OST0001 complete [ 9142.292302] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 9148.105488] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 9170.773735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9170.891111] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9170.908726] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9170.963661] Lustre: lustre-MDT0000: new disk, initializing [ 9171.005082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9171.011630] Lustre: Skipped 1 previous similar message [ 9171.021627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9173.304156] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9178.951955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9178.996824] Lustre: 240969:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9179.028188] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9179.030732] Lustre: Skipped 1 previous similar message [ 9179.071816] Lustre: lustre-MDT0001: new disk, initializing [ 9179.150089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9179.154433] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9181.342349] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9186.492902] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9189.482621] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9193.951719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9194.159375] Lustre: 242936:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9195.241172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9195.249488] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9195.318191] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9197.552926] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9203.149084] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9205.326740] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9205.443229] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9207.909189] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9208.021645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9210.855626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9210.856752] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9210.860966] LustreError: Skipped 1 previous similar message [ 9210.861973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9210.866840] Lustre: Skipped 1 previous similar message [ 9210.872819] Lustre: Skipped 1 previous similar message [ 9215.159882] Lustre: server umount lustre-OST0000 complete [ 9220.068629] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9220.072803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9220.079973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9231.328304] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9231.330771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9231.336051] Lustre: Skipped 5 previous similar messages [ 9231.408169] Lustre: server umount lustre-MDT0000 complete [ 9232.925359] LustreError: 240959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789682597 with bad export cookie 14973775701575070644 [ 9232.928143] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9232.930082] LustreError: 240959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9233.060120] Lustre: server umount lustre-MDT0001 complete [ 9236.730567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9236.994355] LustreError: 245149:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9237.039880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9237.043263] Lustre: Skipped 2 previous similar messages [ 9239.020596] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9242.082328] LustreError: 245149:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9242.093374] LustreError: 245149:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9243.310037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9245.640096] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9249.805419] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9253.130978] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9257.903335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9261.834809] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9265.803406] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9411.041532] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9411.046416] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9411.053212] Lustre: Skipped 1 previous similar message [ 9411.055143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9422.816172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9422.886332] Lustre: server umount lustre-OST0000 complete [ 9427.426283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9427.432156] Lustre: Skipped 1 previous similar message [ 9430.703828] Lustre: server umount lustre-MDT0000 complete [ 9432.292640] LustreError: 245130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789682796 with bad export cookie 14973775701575071526 [ 9432.296377] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9432.300287] LustreError: 245130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9432.435184] Lustre: server umount lustre-MDT0001 complete [ 9530.339324] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 9531.788143] Key type lgssc unregistered [ 9531.964527] LNet: 248751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9531.969418] LNetError: 248751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9531.978434] LNet: Removed LNI 192.168.201.132@tcp [ 9532.394152] Key type .llcrypt unregistered [ 9532.395714] Key type ._llcrypt unregistered [ 9543.393476] Key type ._llcrypt registered [ 9543.394759] Key type .llcrypt registered [ 9543.452142] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 9552.011093] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 9552.545174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9552.558693] alg: No test for adler32 (adler32-zlib) [ 9553.440593] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 9553.568446] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 9555.160167] Key type lgssc registered [ 9555.648424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9576.428160] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 9580.680984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9580.689866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9581.795437] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9581.806467] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9581.844167] Lustre: lustre-MDT0000: new disk, initializing [ 9581.876635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9581.885910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9583.751479] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9588.948681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9589.008911] Lustre: 252790:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9589.037556] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9589.039820] Lustre: Skipped 1 previous similar message [ 9589.078349] Lustre: lustre-MDT0001: new disk, initializing [ 9589.121789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9589.150036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9589.153517] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9591.038242] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9593.963770] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9597.383942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9597.529213] Lustre: lustre-OST0000: new disk, initializing [ 9597.531565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9597.535092] Lustre: 254422:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9597.604874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9597.715158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9597.719399] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9597.740356] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 9600.850496] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9606.559900] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9606.675486] Lustre: lustre-OST0001: new disk, initializing [ 9606.677808] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9606.681273] Lustre: 255292:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9606.757256] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9609.962401] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9616.408459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9616.411797] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9616.458899] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9617.077406] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9622.581933] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9630.183399] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9630.190623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9630.198985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9631.715791] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9631.716197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9631.719935] Lustre: Skipped 1 previous similar message [ 9631.723192] Lustre: Skipped 3 previous similar messages [ 9633.364250] Lustre: server umount lustre-MDT0000 complete [ 9635.048678] LustreError: 252784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789682999 with bad export cookie 10949058383644850208 [ 9635.052385] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9635.057903] LustreError: 252784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9635.212865] Lustre: server umount lustre-MDT0001 complete [ 9647.495331] Lustre: server umount lustre-OST0000 complete [ 9659.454660] Lustre: server umount lustre-OST0001 complete [ 9665.426518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9669.525641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9673.108529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9676.473390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9685.283803] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 9689.556892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9689.567589] Lustre: lustre-MDT0000: reset Object Index mappings [ 9689.836573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9691.451595] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9695.210955] LustreError: 260294:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9695.235183] LustreError: 260294:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9696.210832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9696.216180] Lustre: lustre-MDT0001: reset Object Index mappings [ 9698.587454] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9701.179659] Lustre: 261437:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9709.914730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9710.332684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9710.341972] Lustre: Skipped 1 previous similar message [ 9715.433786] LustreError: 261792:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9715.450226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 9717.055141] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9720.822684] LustreError: 262056:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9726.955781] LustreError: 261793:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9726.977694] LustreError: 261793:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9728.107710] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9728.310743] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9733.503157] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9733.674813] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9743.229834] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9746.658786] Lustre: 263308:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9752.548754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9752.556755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9752.564880] Lustre: Skipped 1 previous similar message [ 9752.569173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9753.257860] Lustre: server umount lustre-MDT0000 complete [ 9754.081499] LustreError: 260294:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9754.108357] LustreError: 260294:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9756.418390] LustreError: 260274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789683120 with bad export cookie 10949058383644856984 [ 9756.425297] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9756.431458] LustreError: 260274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9756.863847] Lustre: server umount lustre-MDT0001 complete [ 9771.193340] Lustre: server umount lustre-OST0000 complete [ 9774.871319] Lustre: 250302:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789683123/real 1789683123] req@ffff97db0ce77480 x1876618554490112/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789683139 ref 1 fl Rpc:EXNQr/200/ffffffff rc -5/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9785.303033] Lustre: server umount lustre-OST0001 complete [ 9801.891427] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 18:12:44 (1789683164) [ 9815.802415] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 9824.514880] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 9853.104821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9853.331392] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9853.352325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9853.409924] Lustre: lustre-MDT0000: new disk, initializing [ 9853.477066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9853.490328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9856.797848] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9864.539347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9864.645163] Lustre: 267563:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9864.723070] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9864.732904] Lustre: Skipped 1 previous similar message [ 9864.844437] Lustre: lustre-MDT0001: new disk, initializing [ 9864.967976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9864.984154] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9868.485656] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9876.866323] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9883.656433] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9897.394980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9897.633465] Lustre: lustre-OST0000: new disk, initializing [ 9897.636680] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9897.641409] Lustre: 269535:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9898.860448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9898.868715] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9899.100938] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9913.935261] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [ 9933.574691] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9941.146582] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9941.448301] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9947.080723] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9947.298921] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9950.177811] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9950.180954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9950.214029] Lustre: Skipped 3 previous similar messages [ 9950.231556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9955.304403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9955.316453] Lustre: Skipped 2 previous similar messages [ 9955.855796] Lustre: server umount lustre-OST0000 complete [ 9962.468731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9962.478349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9962.506920] Lustre: Skipped 1 previous similar message [ 9965.539977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9965.552981] Lustre: Skipped 2 previous similar messages [ 9966.981957] Lustre: server umount lustre-MDT0000 complete [ 9970.660667] LustreError: 268466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9970.680988] LustreError: 268466:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9971.745326] LustreError: 268358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789683336 with bad export cookie 10949058383644859077 [ 9971.752381] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9971.758726] LustreError: 268358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9972.163915] Lustre: server umount lustre-MDT0001 complete [ 9996.203272] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [10009.300588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10009.948116] LustreError: 272472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10010.009641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10010.014422] Lustre: Skipped 2 previous similar messages [10015.607632] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10027.561294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10034.527453] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10038.194249] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10047.196190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10056.173376] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10067.309183] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10067.712105] Lustre: lustre-OST0001: new disk, initializing [10067.731379] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10067.750974] Lustre: 274781:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10069.421258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10069.436586] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10069.565235] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10076.136707] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10088.266191] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10093.037492] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10099.810725] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10099.838150] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10115.046148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10115.058089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10115.080656] Lustre: Skipped 1 previous similar message [10115.085963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10117.192848] Lustre: server umount lustre-MDT0000 complete [10120.687118] LustreError: 272473:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10120.705205] LustreError: 272473:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [10121.513591] LustreError: 272455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789683485 with bad export cookie 10949058383644859959 [10121.518484] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10121.526870] LustreError: 272455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10121.925246] Lustre: server umount lustre-MDT0001 complete [10136.172244] Lustre: server umount lustre-OST0000 complete [10151.404915] Lustre: server umount lustre-OST0001 complete [10173.763548] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [10186.640959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10187.202976] LustreError: 278129:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10187.345298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10187.351921] Lustre: Skipped 3 previous similar messages [10192.785171] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10202.502751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10208.742809] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10212.216797] Lustre: 279271:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10220.040713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10227.728710] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10231.881995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10237.156954] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10242.561710] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10245.410592] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10254.089160] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10258.309961] Lustre: 281149:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10264.566432] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10264.696566] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10279.924255] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 18:20:42 (1789683642) [10298.866824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10298.873597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10298.889266] Lustre: Skipped 5 previous similar messages [10298.899991] Lustre: Skipped 6 previous similar messages [10302.560760] Lustre: server umount lustre-MDT0000 complete [10307.203310] LustreError: 278110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789683671 with bad export cookie 10949058383644866826 [10307.228110] LustreError: 278110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10307.230791] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10314.078944] Lustre: server umount lustre-MDT0001 complete [10326.243963] Lustre: server umount lustre-OST0000 complete [10331.189683] Lustre: server umount lustre-OST0001 complete [10340.866799] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [10354.157801] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [10420.435782] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [10434.848347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10435.299612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10435.339304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10435.428770] Lustre: lustre-MDT0000: new disk, initializing [10435.570376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10442.419524] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10455.308536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10455.498948] Lustre: 285700:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10456.041270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10456.047326] Lustre: Skipped 4 previous similar messages [10456.129674] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10462.827426] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10468.981291] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10481.478662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10482.204965] Lustre: 287336:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10483.827871] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10491.432082] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10504.748759] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10505.096546] Lustre: 288207:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10506.727199] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10506.741445] Lustre: Skipped 1 previous similar message [10506.836058] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10513.224856] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10524.921517] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10548.944956] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 18:25:10 (1789683910) [10552.170926] Lustre: server umount lustre-MDT0000 complete [10552.834547] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10552.838091] LustreError: 285708:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10552.849175] Lustre: Skipped 6 previous similar messages [10552.881795] LustreError: 285708:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [10557.296578] LustreError: 285695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789683921 with bad export cookie 10949058383644874302 [10557.303458] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10557.306655] LustreError: 285695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10557.941880] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10557.947767] Lustre: Skipped 5 previous similar messages [10560.089536] Lustre: server umount lustre-MDT0001 complete [10564.921738] Lustre: server umount lustre-OST0000 complete [10568.942154] Lustre: server umount lustre-OST0001 complete [10580.059218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10586.864729] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10598.005471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10605.556739] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10613.825984] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10621.075150] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10628.976554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10639.339885] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10649.299449] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10673.467950] loop1: detected capacity change from 0 to 204800000 [10673.759292] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10673.788888] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10673.820170] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10680.881045] loop1: detected capacity change from 0 to 204800000 [10689.113861] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10689.294647] LustreError: 290445:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs13093: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10689.339890] LustreError: 290445:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs13093 [10689.354117] LustreError: 290445:0:(mgs_handler.c:517:mgs_target_reg()) fs13093-MDT0001: Error checking target index: rc = -13 [10689.373868] LustreError: fs13093-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10689.386472] LustreError: 294182:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10689.408478] LustreError: 294182:0:(tgt_mount.c:2128:server_put_super()) no obd fs13093-MDT0001 [10689.421472] LustreError: 294182:0:(tgt_mount.c:116:server_deregister_mount()) fs13093-MDT0001 not registered [10689.507242] Lustre: server umount fs13093-MDT0001 complete [10689.514599] LustreError: 294182:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10701.732194] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10701.831806] Lustre: Setting parameter fs13093-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs13093-MDT0001 [10701.839729] Lustre: Skipped 1 previous similar message [10701.889225] Lustre: srv-fs13093-MDT0001: No data found on store. Initialize space. [10701.894290] Lustre: Skipped 4 previous similar messages [10701.986840] Lustre: fs13093-MDT0001: new disk, initializing [10701.988962] Lustre: Skipped 3 previous similar messages [10702.143937] Lustre: srv-fs13093-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10702.156836] Lustre: 294649:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13093-MDT0001-mdtlov: can't create new llog: rc = -115 [10702.181229] LustreError: 294649:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13093-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10709.468116] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10717.764475] Lustre: server umount fs13093-MDT0001 complete [10721.782460] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10733.108357] loop1: detected capacity change from 0 to 204800000 [10739.408529] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10739.696224] Lustre: 296217:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13093-MDT0001-mdtlov: can't create new llog: rc = -115 [10739.704373] LustreError: 296217:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13093-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10744.514798] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10750.941172] Lustre: server umount fs13093-MDT0001 complete [10767.329042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10767.336757] LustreError: Skipped 1 previous similar message [10767.338493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10767.344125] Lustre: Skipped 2 previous similar messages [10767.347156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10770.350566] Lustre: server umount lustre-MDT0000 complete [10780.624361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10780.848466] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10786.298039] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10786.768984] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10795.465830] loop1: detected capacity change from 0 to 204800000 [10801.039651] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10801.358177] Lustre: 298665:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13093-MDT0001-mdtlov: can't create new llog: rc = -115 [10801.370632] LustreError: 298665:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13093-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10805.977093] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10811.251545] Lustre: server umount fs13093-MDT0001 complete [10811.887403] LustreError: 292291:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10811.933809] LustreError: 292291:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 109 previous similar messages [10837.482879] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10840.512077] Lustre: server umount lustre-OST0000 complete [10850.395157] Lustre: server umount lustre-MDT0000 complete [10854.246768] LustreError: 290487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789684218 with bad export cookie 10949058383644876983 [10854.249482] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10854.281493] LustreError: 290487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10854.676914] Lustre: server umount lustre-MDT0001 complete [10866.957623] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [10869.627904] Key type lgssc unregistered [10869.955774] LNet: 301006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10869.969298] LNetError: 301006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10869.999399] LNet: Removed LNI 192.168.201.132@tcp [10871.389277] Key type .llcrypt unregistered [10871.392393] Key type ._llcrypt unregistered [10897.538840] Key type ._llcrypt registered [10897.541258] Key type .llcrypt registered [10897.702186] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [10898.556402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10898.578213] alg: No test for adler32 (adler32-zlib) [10899.721630] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [10900.180293] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [10901.936608] Key type lgssc registered [10903.572331] Lustre: Echo OBD driver; http://www.lustre.org/ [10912.896640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10912.912148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10914.292424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10918.067140] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10924.516760] LustreError: 302278:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10924.531163] LustreError: 302278:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10929.225671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10929.638712] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10929.885800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10936.621577] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10944.893616] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10951.845907] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10959.058510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10959.422340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10965.659964] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [10973.843544] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10974.755179] LustreError: 304106:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10979.819710] LustreError: 304108:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10983.914337] LustreError: 304106:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10989.037278] LustreError: 304106:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10989.046847] LustreError: 304106:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10999.285310] LustreError: 304778:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10999.305340] LustreError: 304778:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11004.509442] loop1: detected capacity change from 0 to 204800000 [11004.917449] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11004.928389] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11004.942263] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11011.696490] loop1: detected capacity change from 0 to 204800000 [11016.697842] LustreError: 304778:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11016.711264] LustreError: 304778:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [11018.832321] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11019.014048] Lustre: fs24811-OST0000: new disk, initializing [11019.028669] Lustre: srv-fs24811-OST0000: No data found on store. Initialize space. [11019.035726] Lustre: 306034:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11019.120804] Lustre: fs24811-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11025.587704] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11036.252969] Lustre: server umount fs24811-OST0000 complete [11050.220293] loop1: detected capacity change from 0 to 204800000 [11052.541144] LustreError: 304778:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11052.562023] LustreError: 304778:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 20 previous similar messages [11053.570651] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11056.435849] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11056.676447] Lustre: fs24811-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11058.666108] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11063.807226] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11073.751103] loop2: detected capacity change from 0 to 204800000 [11073.997382] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11074.006470] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11074.028151] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11080.261966] loop2: detected capacity change from 0 to 204800000 [11086.367500] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11086.441310] LustreError: fs24811-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11086.450547] LustreError: fs24811-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11086.458707] LustreError: 309076:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [11086.466796] LustreError: 309076:0:(tgt_mount.c:2128:server_put_super()) no obd fs24811-OST0001 [11086.475143] LustreError: 309076:0:(tgt_mount.c:116:server_deregister_mount()) fs24811-OST0001 not registered [11086.561501] Lustre: server umount fs24811-OST0001 complete [11086.565910] LustreError: 309076:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11098.177020] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11098.529927] Lustre: fs24811-OST0001: new disk, initializing [11098.533238] Lustre: srv-fs24811-OST0001: No data found on store. Initialize space. [11098.541900] Lustre: 309627:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11098.659668] Lustre: fs24811-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11106.122963] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11114.784772] Lustre: server umount fs24811-OST0000 complete [11119.117883] LustreError: 304106:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11119.158153] LustreError: 304106:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [11125.958356] Lustre: server umount fs24811-OST0001 complete [11140.024529] loop1: detected capacity change from 0 to 204800000 [11146.080470] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11146.306865] Lustre: fs24811-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11154.114531] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11161.595370] loop2: detected capacity change from 0 to 204800000 [11168.130720] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11175.493991] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11185.473597] Lustre: server umount fs24811-OST0000 complete [11196.972604] Lustre: server umount fs24811-OST0001 complete [11207.152485] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11212.261457] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11212.265893] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11212.284675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11212.302063] Lustre: Skipped 1 previous similar message [11216.761825] Lustre: server umount lustre-OST0000 complete [11221.488377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11221.497938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11221.533872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11221.546530] Lustre: Skipped 1 previous similar message [11222.506046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11227.454773] Lustre: server umount lustre-MDT0000 complete [11231.125450] LustreError: 310376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789684595 with bad export cookie 3463786473877851120 [11231.140927] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11231.406801] Lustre: server umount lustre-MDT0001 complete [11244.542138] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [11247.705711] Key type lgssc unregistered [11248.005310] LNet: 315550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11248.012890] LNetError: 315550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11248.041896] LNet: Removed LNI 192.168.201.132@tcp [11249.077166] Key type .llcrypt unregistered [11249.078934] Key type ._llcrypt unregistered [11280.516537] Key type ._llcrypt registered [11280.517666] Key type .llcrypt registered [11280.592490] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [11292.790544] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [11293.476935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11293.504319] alg: No test for adler32 (adler32-zlib) [11294.565935] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [11294.748193] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [11296.400223] Key type lgssc registered [11297.365515] Lustre: Echo OBD driver; http://www.lustre.org/ [11334.715569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11334.747437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11336.057088] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11336.107424] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11336.212206] Lustre: lustre-MDT0000: new disk, initializing [11336.336089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11336.365650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11341.688426] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11353.470181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11353.670818] Lustre: 319385:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11353.763755] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11353.769779] Lustre: Skipped 1 previous similar message [11353.854270] Lustre: lustre-MDT0001: new disk, initializing [11353.976421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11354.027697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11354.040422] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11359.253309] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11369.293803] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11374.786474] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11381.359500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11381.750758] Lustre: lustre-OST0000: new disk, initializing [11381.754273] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11381.760948] Lustre: 321357:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11381.842948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11388.404890] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11391.102486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11391.116863] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11391.258927] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11398.854258] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11400.684666] Lustre: Mounted lustre-client - version 2.17.58_39_g9373f0b [11401.613976] Lustre: Unmounted lustre-client [11410.915240] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11410.920653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11410.941118] Lustre: Skipped 1 previous similar message [11416.046117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11416.047941] Lustre: Skipped 1 previous similar message [11416.527776] Lustre: server umount lustre-OST0000 complete [11420.641852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11420.644115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11420.661464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11428.121231] Lustre: server umount lustre-MDT0000 complete [11431.397245] LustreError: 319396:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11431.409271] LustreError: 319396:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11431.493084] LustreError: 319380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789684795 with bad export cookie 6172620568545299375 [11431.498813] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11431.504835] LustreError: 319380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11431.812679] Lustre: server umount lustre-MDT0001 complete [11440.753756] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [11443.204186] Key type lgssc unregistered [11450.400602] LNet: 1 local NIs in recovery (showing 1): 192.168.201.132@tcp [11450.656706] LNet: 323515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11450.665700] LNetError: 323515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11531.558819] LNet: Removed LNI 192.168.201.132@tcp [11532.389802] Key type .llcrypt unregistered [11532.391778] Key type ._llcrypt unregistered [11549.640213] Key type ._llcrypt registered [11549.641805] Key type .llcrypt registered [11549.750568] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [11562.673854] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [11563.538145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11563.602134] alg: No test for adler32 (adler32-zlib) [11564.680226] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [11564.863542] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [11566.536271] Key type lgssc registered [11567.513380] Lustre: Echo OBD driver; http://www.lustre.org/ [11589.999331] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 18:42:32 (1789684952) [11595.050428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11595.071877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11596.304608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11596.339697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11596.394080] Lustre: lustre-MDT0000: new disk, initializing [11596.455670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11596.469378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11600.125962] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11608.801436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11608.894994] Lustre: 326723:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11608.946162] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11608.950144] Lustre: Skipped 1 previous similar message [11609.001251] Lustre: lustre-MDT0001: new disk, initializing [11609.062357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11609.106728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11609.117099] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11612.199704] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11619.423736] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11623.608302] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11628.828275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11629.033381] Lustre: lustre-OST0000: new disk, initializing [11629.035272] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11629.038633] Lustre: 328690:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11629.111280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11634.179713] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11636.288840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11636.307707] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11636.390711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11642.598534] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11657.187358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11657.190401] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11657.198229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11657.207133] Lustre: Skipped 1 previous similar message [11660.236626] Lustre: server umount lustre-OST0000 complete [11665.382445] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11665.393237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11665.405037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11665.407813] Lustre: Skipped 1 previous similar message [11667.430920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11667.431548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11667.463609] Lustre: Skipped 1 previous similar message [11669.475579] Lustre: server umount lustre-MDT0000 complete [11672.310919] LustreError: 326715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685036 with bad export cookie 1960551668469875622 [11672.313875] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11672.316357] LustreError: 326715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11672.544146] Lustre: server umount lustre-MDT0001 complete [11682.235350] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [11684.625369] Key type lgssc unregistered [11684.871226] LNet: 330597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11684.876206] LNetError: 330597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11684.895422] LNet: Removed LNI 192.168.201.132@tcp [11685.576510] Key type .llcrypt unregistered [11685.578141] Key type ._llcrypt unregistered [11702.395760] Key type ._llcrypt registered [11702.398571] Key type .llcrypt registered [11702.491988] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [11703.421510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11703.508519] alg: No test for adler32 (adler32-zlib) [11704.532844] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [11704.721754] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [11706.352219] Key type lgssc registered [11707.223775] Lustre: Echo OBD driver; http://www.lustre.org/ [11713.561082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11713.594731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11714.938614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11718.618874] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11725.289311] LustreError: 331845:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11725.304168] LustreError: 331845:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11726.644106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11726.976348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11730.136624] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11735.870353] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11740.712568] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11746.573948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11746.909783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11752.909611] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11759.675512] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11773.338886] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11775.181119] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11780.640621] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.32@tcp (stopping) [11781.092149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11781.114285] Lustre: Skipped 2 previous similar messages [11785.721923] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.32@tcp (stopping) [11785.728613] Lustre: Skipped 3 previous similar messages [11790.848874] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.32@tcp (stopping) [11790.866998] Lustre: Skipped 3 previous similar messages [11793.376476] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11793.583181] Lustre: server umount lustre-MDT0000 complete [11795.977617] LustreError: 331841:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11797.475831] LustreError: 331827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685161 with bad export cookie 5713562535772353675 [11797.487655] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11797.490703] LustreError: 331827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11801.082289] LustreError: 331842:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11801.117060] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.32@tcp (stopping) [11801.122020] LustreError: 331842:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11801.154910] Lustre: Skipped 3 previous similar messages [11801.574436] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11803.956671] Lustre: server umount lustre-MDT0001 complete [11807.905590] Lustre: server umount lustre-OST0000 complete [11816.088449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11819.680597] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11822.067820] Lustre: 336025:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.132@tcp [11823.923586] Lustre: server umount MGS complete [11832.079487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11832.401729] LustreError: 336598:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11832.415691] LustreError: 336598:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11832.475135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11836.752367] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11837.922282] LustreError: 336599:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11844.776784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11845.144338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11849.471837] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11857.041287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11857.413310] LustreError: 338031:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11857.440943] LustreError: 338031:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11857.768905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11866.763492] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [11878.965797] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11881.193229] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11888.965099] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11891.096743] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11898.861714] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11898.885429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11898.911109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11898.922271] Lustre: Skipped 2 previous similar messages [11900.915561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11900.924829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11904.215825] Lustre: server umount lustre-OST0000 complete [11911.137914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11911.145347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11911.175645] Lustre: Skipped 1 previous similar message [11911.204504] Lustre: Skipped 3 previous similar messages [11916.541309] Lustre: server umount lustre-MDT0000 complete [11921.381255] LustreError: 336599:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11921.422263] LustreError: 336599:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11921.524469] LustreError: 336580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685285 with bad export cookie 5713562535772354515 [11921.530682] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11921.553602] LustreError: 336580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11922.788935] Lustre: server umount lustre-MDT0001 complete [11938.644346] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [11941.984580] Key type lgssc unregistered [11942.259802] LNet: 340034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11942.268830] LNetError: 340034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11943.346073] LNet: Removed LNI 192.168.201.132@tcp [11944.406973] Key type .llcrypt unregistered [11944.409362] Key type ._llcrypt unregistered [11965.892454] Key type ._llcrypt registered [11965.894844] Key type .llcrypt registered [11966.037816] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [11984.587745] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [11985.616418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11985.676515] alg: No test for adler32 (adler32-zlib) [11987.095678] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [11987.347056] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [11989.032769] Key type lgssc registered [11990.053806] Lustre: Echo OBD driver; http://www.lustre.org/ [12024.245722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12024.267929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12025.610346] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12025.675558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12025.798549] Lustre: lustre-MDT0000: new disk, initializing [12025.885046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12025.908237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12030.805554] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [12041.191711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12041.348830] Lustre: 343195:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12041.410361] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12041.413916] Lustre: Skipped 1 previous similar message [12041.495798] Lustre: lustre-MDT0001: new disk, initializing [12041.628727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12041.669617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12041.680354] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12046.387690] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [12056.462089] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12062.274392] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12068.886267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12069.204411] Lustre: lustre-OST0000: new disk, initializing [12069.212749] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12069.220606] Lustre: 345162:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12069.308579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12075.501140] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [12076.657840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12076.670169] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12076.901109] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12085.205724] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12090.623952] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12090.847654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12095.921415] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12096.113237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12102.638123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12102.639346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12102.647248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12102.672935] Lustre: Skipped 1 previous similar message [12107.758474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12107.767300] Lustre: Skipped 2 previous similar messages [12112.352127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12112.632738] Lustre: server umount lustre-OST0000 complete [12117.986438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12117.987235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12117.996495] Lustre: Skipped 1 previous similar message [12118.008287] Lustre: Skipped 1 previous similar message [12123.110328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12123.119556] Lustre: Skipped 1 previous similar message [12128.225340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12128.240522] Lustre: Skipped 1 previous similar message [12131.296910] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12131.499396] Lustre: server umount lustre-MDT0000 complete [12135.286046] LustreError: 343186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685499 with bad export cookie 14585273906859972050 [12135.296394] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12135.300436] LustreError: 343186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12135.619794] Lustre: server umount lustre-MDT0001 complete [12145.127723] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 18:51:47 (1789685507) [12162.469670] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [12172.491291] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [12208.069073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12208.426683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12208.461611] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12208.566864] Lustre: lustre-MDT0000: new disk, initializing [12208.691063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12208.735555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12213.489235] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [12223.560556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12223.674984] Lustre: 349923:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12223.725837] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12223.728706] Lustre: Skipped 1 previous similar message [12223.790937] Lustre: lustre-MDT0001: new disk, initializing [12223.882381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12223.913570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12223.928492] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12228.099545] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [12237.380813] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12245.543189] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12257.112464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12257.626255] Lustre: lustre-OST0000: new disk, initializing [12257.633152] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12257.644673] Lustre: 351891:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12257.799303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12267.122842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12267.148476] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12267.329976] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12269.908860] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [12281.677796] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12323.809349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12323.816046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12323.826308] LustreError: Skipped 2 previous similar messages [12323.827037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12323.865095] Lustre: Skipped 1 previous similar message [12326.803413] Lustre: server umount lustre-OST0000 complete [12331.502631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12331.506314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12337.267845] Lustre: server umount lustre-MDT0000 complete [12339.176205] LustreError: 351532:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12339.193555] LustreError: 351532:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12341.197757] LustreError: 349914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685705 with bad export cookie 14585273906859972939 [12341.203992] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12341.212101] LustreError: 349914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12341.823480] Lustre: server umount lustre-MDT0001 complete [12354.102424] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [12357.549773] Key type lgssc unregistered [12357.740805] LNet: 354070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12357.745843] LNetError: 354070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12358.767798] LNet: Removed LNI 192.168.201.132@tcp [12359.595143] Key type .llcrypt unregistered [12359.596427] Key type ._llcrypt unregistered [12480.490641] Key type ._llcrypt registered [12480.496480] Key type .llcrypt registered [12480.664976] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [12480.986448] Key type .llcrypt unregistered [12480.988660] Key type ._llcrypt unregistered [12566.586036] Key type ._llcrypt registered [12566.595920] Key type .llcrypt registered [12566.791130] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [12567.255299] Key type .llcrypt unregistered [12567.256743] Key type ._llcrypt unregistered [12600.777577] Key type ._llcrypt registered [12600.781412] Key type .llcrypt registered [12600.920505] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [12601.256472] Key type .llcrypt unregistered [12601.261326] Key type ._llcrypt unregistered [12636.102366] Key type ._llcrypt registered [12636.104821] Key type .llcrypt registered [12636.277853] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [12636.666547] Key type .llcrypt unregistered [12636.670866] Key type ._llcrypt unregistered [12727.339934] Key type ._llcrypt registered [12727.343691] Key type .llcrypt registered [12727.456752] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [12727.871451] Key type .llcrypt unregistered [12727.874715] Key type ._llcrypt unregistered [12767.927259] Key type ._llcrypt registered [12767.930269] Key type .llcrypt registered [12768.172482] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [12768.978378] Key type .llcrypt unregistered [12768.994751] Key type ._llcrypt unregistered [12814.390641] Key type ._llcrypt registered [12814.392168] Key type .llcrypt registered [12814.668361] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [12815.014322] Key type .llcrypt unregistered [12815.017966] Key type ._llcrypt unregistered [12856.860633] Key type ._llcrypt registered [12856.862788] Key type .llcrypt registered [12857.139472] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [12857.660296] Key type .llcrypt unregistered [12857.662109] Key type ._llcrypt unregistered [12892.028220] Key type ._llcrypt registered [12892.031776] Key type .llcrypt registered [12892.135550] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [12892.537575] Key type .llcrypt unregistered [12892.542296] Key type ._llcrypt unregistered [12925.566330] Key type ._llcrypt registered [12925.568802] Key type .llcrypt registered [12925.658310] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [12925.952340] Key type .llcrypt unregistered [12925.954066] Key type ._llcrypt unregistered [13001.667204] Key type ._llcrypt registered [13001.669175] Key type .llcrypt registered [13001.772913] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [13002.137397] Key type .llcrypt unregistered [13002.142439] Key type ._llcrypt unregistered [13039.726319] Key type ._llcrypt registered [13039.729361] Key type .llcrypt registered [13039.844911] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [13040.168603] Key type .llcrypt unregistered [13040.172252] Key type ._llcrypt unregistered [13079.170252] Key type ._llcrypt registered [13079.172793] Key type .llcrypt registered [13079.326906] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [13079.848763] Key type .llcrypt unregistered [13079.853532] Key type ._llcrypt unregistered [13146.347249] Key type ._llcrypt registered [13146.348545] Key type .llcrypt registered [13146.455847] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [13164.483164] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [13165.923195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13165.980361] alg: No test for adler32 (adler32-zlib) [13167.289950] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [13167.661660] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [13169.400216] Key type lgssc registered [13170.611217] Lustre: Echo OBD driver; http://www.lustre.org/ [13208.521395] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13208.543922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13209.743986] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13209.777439] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13209.829782] Lustre: tst250fs-MDT0000: new disk, initializing [13209.882227] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13209.897794] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13213.646417] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [13223.172052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13223.320941] Lustre: 374407:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13223.409853] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13223.414040] Lustre: Skipped 1 previous similar message [13223.508970] Lustre: tst250fs-MDT0001: new disk, initializing [13223.582357] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13223.614827] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13223.630434] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13227.302208] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [13231.986701] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13238.080114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13238.312698] Lustre: tst250fs-OST0000: new disk, initializing [13238.314989] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13238.320233] Lustre: 376036:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13238.410762] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13238.755933] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13238.767425] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13238.875759] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13244.441468] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [13254.375337] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13254.561919] Lustre: tst250fs-OST0001: new disk, initializing [13254.566931] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13254.574193] Lustre: 376909:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13254.675227] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13260.606286] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all [13263.978409] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13263.989019] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13264.097534] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13265.840093] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13267.950687] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13267.954228] Lustre: Skipped 2 previous similar messages [13284.843437] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13284.848206] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13284.859536] Lustre: Skipped 3 previous similar messages [13284.869931] Lustre: Skipped 3 previous similar messages [13289.954105] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13289.973783] Lustre: Skipped 3 previous similar messages [13290.760802] Lustre: server umount tst250fs-MDT0000 complete [13294.180814] LustreError: 374397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789686658 with bad export cookie 4284038236787319032 [13294.186128] LustreError: 374397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13294.187043] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13294.619945] Lustre: server umount tst250fs-MDT0001 complete [13308.303632] Lustre: server umount tst250fs-OST0000 complete [13321.684521] Lustre: server umount tst250fs-OST0001 complete [13335.487294] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [13341.075367] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [13362.334353] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 19:12:05 (1789686725) [13363.577369] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13375.448285] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [13382.305990] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [13404.578593] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13114 sec ========= 19:12:47 (1789686767) [13405.549896] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:12:48 (1789686768) === [13407.608792] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:12:50 (1789686770) === [13431.783359] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [13433.537673] Key type lgssc unregistered [13433.745284] LNet: 386028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13433.753896] LNetError: 386028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13434.791606] LNet: Removed LNI 192.168.201.132@tcp [13435.401278] Key type .llcrypt unregistered [13435.404545] Key type ._llcrypt unregistered